CRT monitor style overlay

Software | Web | Coding

Using docker-compose with Django, Nginx and Gunicorn

I host the starcross website on my own server, and value the ability to easily move the site around - experimenting with different linux distros and hardware

There are a number of guides on building a Django oriented docker compose stack. Some are a little overcomplicated for basic setup, others I found very helpful and I have provided some links at the end of this article. I have described the config I ended up using. This will assume some familiarity of hosting Django in a production environment

My key aims are:

  • Maximise portability of the deployment, as this a key advantage of docker
  • Use official docker images where possible for stability and longevity
  • Integrate automatic certificate configuration from Let’s Encrypt (certbot)
  • Use minimal scripting

I am using docker-compose to manage the docker containers and volumes

The docker containers used are:

  • Nginx - Front end to dispatch Gunicorn requests and serve static / media data
  • Python - Containing the Django deployment, and all Python dependencies including the application server (Gunicorn)
  • Postgres - Database for the Django site

I have used a .env file for docker-compose to store configuration information such as usernames and password. This is not committed to the repository, but I have described what variables have been used

I used docker volumes instead of bind mounts. This seems to be the preferred mechanism with the promise of better performance. I am not convinced they are easier to back up or migrate, however copying the data directory worked well enough. There are some pretty funky commands and scripts needed to manage volumes; it would be nice to see something simpler. 

Volumes are used for the Django static and media directories. These correspond to the paths specified in the settings.py

- django-static:/starcross/static
- django-media:/starcross/media

nginx

For nginx I used nginx-certbot to take care of my Let’s Encrypt certificates. This image automatically runs certbot as needed, and uses a volume to store the certificates. It will do the 90 day certificate renewal and port 80->443 redirects automatically. It generally works well for me, although some of the functionality is a bit magical.

Here is the docker-compose entry:

services:
  nginx:
    restart: always
    build: ./nginx/
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - django-static:/starcross/static
      - django-media:/starcross/media
      - www-certs:/etc/letsencrypt
    env_file: .env
    links:
      - starcross:starcross

Nginx-certbot requires an environmental variable is specified for the certificate to be issued

CERTBOT_EMAIL=user@domain.com

Here is the dockerfile. It copies any config files from a local conf.d into the image, which will be picked up by nginx.

FROM staticfloat/nginx-certbot:latest
RUN rm -rf /etc/nginx/user.conf.d/*
COPY conf.d/ /etc/nginx/user.conf.d/

In the conf.d folder, starcross_nginx.conf describes the website configuration -

server {

    listen 443 ssl;
    server_name starcross.dev www.starcross.dev localhost 127.0.0.1;

    ssl_certificate     /etc/letsencrypt/live/starcross.dev/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/starcross.dev/privkey.pem;

    charset utf-8;

    location /static {
        alias /starcross/static;
    }

    location /media {
        alias /starcross/media;
    }

    location / {
        proxy_pass http://starcross:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

The ssl_certificate directives are required by nginx when using ssl and their paths are specific to certbot.

You may find it easier to use a standard nginx image to start, before dealing with the extra complication of certificates and additional ports.

The static and media location blocks enable hosting of the static Django content, and should be familiar if you have set up Django for production before. 

Finally all traffic is redirected to port 80 as a reverse proxy. The ‘starcross’ host refers to the name python/django container in the docker-compose.yml, and will be available in docker’s internal network.

Python (Django)

For the Django container I have used the slim Python image. I chose not to use the more compact alpine based image due to reports of obscure issues sometimes arising. 

The dockerfile uses pip to install the packages in my requirements.txt, which includes Django, Gunicorn and all other dependencies needed by the site

FROM python:3.8-slim
WORKDIR /starcross
COPY requirements.txt requirements.txt
RUN python -m pip install -r requirements.txt
COPY . .

The final copy statement places the whole django project in the image ready for gunicorn to run. 

starcross:
    depends_on:
        - postgres
    build: ./starcross/
    volumes:
      - django-static:/starcross/static
      - django-media:/starcross/media
    env_file: .env
    command: sh -c "python manage.py collectstatic --noinput &&
                    gunicorn starcross.wsgi:application --workers=2 --threads=4 --worker-class=gthread
                                                        --bind :80 --worker-tmp-dir /dev/shm"

The static and media volumes allow site data to persist each time the container is run (for example the images in my gallery). Two commands are run in succession when the image starts - the django collectstatic command and then guincorn is started. 

python manage.py collectstatic --noinput

I find it convenient to run collectstatic as I would certainly forget each time I make file changes. This is useful for achieving continuous integration in testing environments as well. More complex updates like model changes will require manual intervention however.

For a wgsi server I used Gunicorn. There are some interesting and conflicting benchmarks available for wsgi server performance. I am interested in single (not concurrent) request time and found gunicorn held an edge here over uwsgi, especially for more cpu intensive pages like those in starcross gallery. For most uses I think any perfomance difference will not be noticable.

gunicorn starcross.wsgi:application --workers=2 --threads=4 --worker-class=gthread
                                    --bind :80 --worker-tmp-dir /dev/shm

The gunicorn command contains options specific to my hardware - 2 workers and 4 threads as recommended for a dual core system. You may add your own customisations here - or you could use uwsgi instead.

The .env file contains PYTHONUNBUFFERED=true to force messages to be printed to the docker log immediately.  It can also be used for Django specific settings without changing the settings.py file for each deployment. For example it can be used to set a unique SECRET_KEY with a corresponding os.environ['SECRET_KEY'] in the settings.py.

SECRET_KEY=mysecret
PYTHONUNBUFFERED=true

You can use Python to generate a new secret key

import secrets
print(secrets.token_urlsafe())

Postgres database

For the database I am using a standard Postgres image with a single volume

postgres:
  restart: always
  image: postgres:9.6.18
  env_file: .env
  volumes:
    - pgdata:/var/lib/postgresql/data/

The .env file contains the database access credentials 

POSTGRES_USER=user
POSTGRES_PASSWORD=password

GitHub

All files are available on Github. I have abridged parts here for brevity, but there should be enough for a minimal setup. My setup includes both the starcross django site and my wife’s wordpress site, so the docker files also include a wordpress and mysql image.

Troubleshooting

The setup can be configured without nginx first. By adding port mappings to docker-compose.yml, individual containers can be tested, such as Django and Postgres.

Django can be switched to debug mode to get more meaningful error messages

Be aware of Docker's behaviour when starting volumes linked to existing directories already containing data (e.g. your static and media directories). Data will be copied to the volume automatically, so you may want to remove these existing directories in production.

Useful resources

Add a comment

captcha

Add one to each digit. 9 becomes 0!