Here at Liquidstate HQ, we use the Jenkins continuous integration server for a wide variety of tasks:

  • Web front end for various scripts
  • Scheduling odd-jobs to run regularly
  • Running various tasks when changes are made to our local Gogs git repositories.

Unfortunately, it seems getting anything but a non-trivial installation is overly involved, particularly if you wish to have Jenkins behind a proxy in a sub-directory. If you've landed hear, I'm guessing you have hit the dreaded “It appears that your reverse proxy set up is broken” issue. I'll further assume that all your usual sources of info (including the Jenkins documentation) are polluted with various "fixes" that are incomplete, obsolete or based on a poor understanding of web protocols and the function of reverse proxies.

In this post, we'll take a look at how we host Jenkins here at Liquidstate HQ, with an aim to sharing our tried and tested setup. At a high level, our solution:

Hosting Jenkins

We have a centralised Docker runtime environment that hosts various Docker containers, including Jenkins. Our docker-compose.yml file for Jenkins looks like this:

version: '2'
services:
  master:
    build: _docker-master
    environment:
    - JAVA_OPTS="-Xmx4096m"
    ports:
    - "8080:8080"
    - "50000:50000"
    volumes:
    - ./data:/var/jenkins_home
    restart: always

Note that the Jenkins web interface is configured to run on the default port 8080 and the slaves will communicate over port 50000.

You'll notice that rather than use an image, we instead build one from a sub-directory called _docker-master. Inside that directory, we have a simple Dockerfile that customises the standard jenkins/jenkins:lts-alpine release to suit our environment - e.g. baking in our own selection of plugins, groovy scripts, etc.

FROM jenkins/jenkins:lts-alpine

# Change URL for use behind a reserve proxy
ENV JENKINS_OPTS "--prefix=/jenkins"

# Install init.d groovy scripts
COPY ./init.groovy.d/executors.groovy       /usr/share/jenkins/ref/init.groovy.d/executors.groovy
COPY ./init.groovy.d/load-properties.groovy /usr/share/jenkins/ref/init.groovy.d/load-properties.groovy

# Install plguins
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

# Avoid banner prompting user to install additional plugins
RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state

By default, Jenkins makes its web interface available at http://localhost:8080/. Note that we're re-configuring the Jenkins URL prefix to /jenkins to change that URL to http://localhost:80808/jenkins/. This is important, as we want our reverse proxy to be able to host several services from their relevant sub-directories.

We have another Docker container dedicated to Nginx. We host this separately as this will provide proxying for all the various services we operate. Our docker-compose.yml file for Nginx looks like this:

web:
  image: nginx:stable-alpine
  volumes:
   - ./default.conf:/etc/nginx/conf.d/default.conf
   - ./html:/usr/share/nginx/html
  ports:
   - "80:80"
  environment:
   - NGINX_HOST=homeserver.liquidstate.net
   - NGINX_PORT=80

As you can see, we're mounting an Nginx configuration file (default.conf) from the local directory into the container as a volume. A simplified version of our default.conf file looks like this:

server {
    listen 80;
    server_name homeserver homeserver.liquidstate.net;

    location /git/ {
        proxy_pass http://homeserver.liquidstate.net:10080/;
    }

    location /jenkins/ {
        # Convert inbound WAN requests for https://domain.tld/jenkins/ to
        # local network requests for http://10.0.0.100:8080/jenkins/
        #
        # Settings mostly from https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy

        # Convert inbound connections for http://.../jenkins/ to http://...:8080/jenkins/;
        proxy_pass         http://homeserver.liquidstate.net:8080;

        # From 
        proxy_redirect     http://homeserver.liquidstate.net:8080 http://homeserver.liquidstate.net;

        proxy_set_header    Host                $host;
        proxy_set_header    X-Real-IP           $remote_addr;
        proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Host    $host:$server_port;
        proxy_set_header    X-Forwarded-Server  $host;
        proxy_set_header    X-Forwarded-Proto   $scheme;

        proxy_max_temp_file_size 0;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;

        # Set maximum upload size
        client_max_body_size       10m;
        client_body_buffer_size    128k;

        # Required for new HTTP-based CLI
        proxy_http_version 1.1;
        proxy_request_buffering off;

        # Sendfile provides no advantages when operating as a proxy
        sendfile off;
    }

}

There's a lot going on in this config, but hopefully you can see that we're hosting two services, each with their own URL.

Now, let's look at some of the critical parts of this config so we can understand what's going on.

proxy_pass

This directive does the heavy lifting of translating user requests for http://.../jenkins/ to http://...:8080/jenkins/. It achieves this by dynamically re-writing the URI provided by the user, fetching content from the backend server.

proxy_redirect

This directive is mostly likely the one you're missing! In addition to basic rewriting of incoming requests, Jenkins expects the reverse proxy to also rewrite any Location: headers to reflect the user-facing URL, rather than its own internal address. So, in our case, we want Nginx to rewrite any Location: or Refresh: headers from http://homeserver.liquidstate.net:8080 to http://homeserver.liquidstate.net.

proxy_set_header

We use the proxy_set_header to set some of the common headers associated with proxying that can be useful for logging/debugging. You can read more about these on Wikipedia.

proxy_http_version

By default, Nginx will use HTTP protocol version 1.0. However, the latest version of Jenkins (and specifically the new HTTP-based CLI requires version 1.1. This directive instructs Nginx to use version 1.1 of the HTTP protocol when talking to Jenkins.

proxy_request_buffering off

When buffering is enabled, the entire request body is read from the client before sending the request to a proxied server. When buffering is disabled, the request body is sent to the proxied server immediately as it is received. Jenkins "streams" data to the user in various parts of the interface, so we want to deliver this to the user as it happens, rather than waiting for the full request to be completed.

Sendfile

Nginx initial fame came from its awesomeness at sending static files. When enabled, sendfile allows Nginx to transfer data from a file descriptor to another directly in kernel space. Unfortunately, there's some restrictions on what file transfers can be accelerated in this fashion. If you’re serving locally stored static files, sendfile is essential to speed your web server. But if you use Nginx as a reverse proxy as we are in this example, it's best to disable it.