Getting started with Django

This repository is an example of how to run a Django app on Kubernetes. It uses the Writing your first Django app Polls application (parts 1 and 2) as the example app to deploy. From here on out, we refer to this app as the 'polls' application.

What we will do ?

In order to deploy this app on Kubernetes we will need to complete the following tasks:

• Create a docker registry access token in order to permit Kubernetes to fetch the docker image we are going to build
• Declare the docker registry access token in variables that will be used in the Continous Integration/Deployment (CI/CD)
• Create a Dockerfile in order to build your Docker image
• Setup Gitlab CI in order to build automaticaly the Docker image at each git push (CI)
• Define Kubernetes manifest in order to deploy
  • A PostgreSQL server with some persistent storage
  • Your Django App()
  • Define a URL in order to access to your application
• Setup Gitlab CI in order to define the Continuous Delivery (CD)

Pre-requisits

On your computer

• Git ust be installed
• Nice to have : a code editor (Atom, Visual Studio code ....)
• Connected to Pasteur VPN (or you won't be able to use all the tools)

Gitlab

If you have never used Gitlab before, you will need to add an ssh key in your profile or define HTTP password in order to fetch the git repository.

Use basic git command :

git add .
git commit -m 'My comment'
git push

Step by step guide

Create Docker Registry Access Token

One of the many Gitlab feature is the ability to host docker images. In order to permit Kubernetes to fetch those images, you need to provide an access token. Please follow the following steps in order to create the access token :

Declare the docker registry access token in variables

We will reuse the previously created access token in order to reuse them in the Continuous Integration pipeline. To do that we will store them in CI variables that we will use during our CI pipeline.

This will prevent from storing credentials in the Git repository. Note that only owners of the project can access to those variables.

Create the Dockerfile

In the root directory of your project create Dockerfile file and fill it like the following :

# https://docs.docker.com/engine/reference/builder/
# FROM refer the image where we start from
# Here we will use a basic python 3 debian image

FROM python:3

# Use the RUN command to pass command to be run
# Here we update the image and add python3 virtualenv

RUN apt-get update && apt-get upgrade -y && apt-get install \
  -y --no-install-recommends python3

# Install pip dependencies
ADD requirements.txt /app/requirements.txt
RUN pip install --default-timeout=100 --upgrade pip && pip install -r /app/requirements.txt

# We add the current content of the git repo in the /app directory
ADD . /app
WORKDIR /app
RUN rm -rf .gitlab-ci.yml Dockerfile README.md img solution
# We use the CMD command to start the gunicorn daemon
# when we start the container.
# Note the $PORT variable, we will need to define it when we start the container
CMD python manage.py runserver 0.0.0.0:$PORT

Setup Gitlab CI

At the root of your git repository create a .gitlab-ci.yml file.

Name it correctly .gitlab-ci.yml and not .gitlab-ci.yaml !

We will use a special docker image which contain the docker binary, we build a docker in a docker :-)

What do we do here ?

• We login on the docker registry enbended in Gitlab (a docker registry is the place where we store docker images to redistribute them)
• We build the docker image and we add a tagto it (actually the id of the git commit)
• We tag the previously builded docker image with a special tag latest (this can be sometime usefull)
• We push the image to the docker registry

stages:
  - build
  - deploy

variables:
  DOCKER_HOST: tcp://localhost:2375/

services:
  - docker:dind

build:
  image: docker:latest
  stage: build
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build -t "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME/polls:${CI_COMMIT_SHORT_SHA}" .
    - docker tag "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME/polls:${CI_COMMIT_SHORT_SHA}" "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME/polls:latest"
    - docker push "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME/polls:${CI_COMMIT_SHORT_SHA}"
  tags:
    - k8s

Save, commit and push; you should be abble to see your first running pipeline

For those not familiar with git you can just do the following in a terminal

git add .
git commit -m 'My comment'
git push

Once succesfully completed, you can see the docker image in the Registrysection on the left pane.

Create manifests yaml files

In order to deploy your application on Kubernetes we need to define how to deploy :

• What is the source of the image ?
• Do I need parameters in order to configure my application ?
• Do I need some storage ?
• What will be it's name ?
• Do I have some "secure" information ? (login, password...)
• What would be it's dns name ? ....

So we are going to create manifest files at the root directory of your git repository and fill it with the following definition.

Keep in mind that yaml formating require that you seperate each declaration with --- line.

PostgreSQL Server

Create the postgresql.yaml file at the root directory of your project

In order to deploy a Postgresql server we need :

• Storage
• Configuration
• Secrets (passwords....)
• Deployment
• Service (Load balancing even if it's for one Pod)

Persistent Volume Claim

As a Docker image is immutable, you may need to define some persistent storage. In the case of a PostgreSQL container we need to persist the data of the database.

We do this using a Persistent Volume Claim.

You can see that we define an accessModesto ReadWriteOnce, this mean that the Persistent Volume will only be accessed by one container.

Insert this in the postgresql.yaml file :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-claim
  labels:
      app: postgresql
spec:
  accessModes:
      - ReadWriteOnce
  resources:
      requests:
        storage: 1Gi

PostgreSQL secret

We are here defining the PostgreSQL basic parameters : username, password and database. This Secret will be reused later in Deployments.

Note: the data have to be base64 encoded. This can be done online or by command line on MacOS or Linux

Insert this in the postgresql.yaml file with --- line before

apiVersion: v1
kind: Secret
metadata:
  name: postgresql-credentials
type: Opaque
data:
  username: cG9sbHNfdXNlcgo=
  password: cG9sbHMK
  database: cG9sbHMK

PostgreSQL Deployment

Insert this in the postgresql.yaml file with --- line before

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgresql
  labels:
    app: postgresql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgresql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: postgresql
        tier: postgreSQL
    spec:
      containers:
      - name: postgresql
        image: postgres:9.6.2-alpine
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: postgresql-credentials
              key: username
        - name: POSTGRES_DB
          valueFrom:
            secretKeyRef:
              name: postgresql-credentials
              key: database
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgresql-credentials
              key: password
        ports:
        - containerPort: 5432
          name: postgresql
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "100m"  
        volumeMounts:
        - name: postgresql
          mountPath: /var/lib/postgresql/data
          subPath: data
      volumes:
      - name: postgresql
        persistentVolumeClaim:
          claimName: postgres-claim
      - name: postgresql-credentials
        secret:
          secretName: postgresql-credentials

PostgreSQL Service

Insert this in the postgresql.yaml file with --- line before

apiVersion: v1
kind: Service
metadata:
  name: postgresql
  labels:
    app: postgresql
spec:
  ports:
    - port: 5432
  selector:
    app: postgresql
    tier: postgreSQL

Django Application

Create polls.yaml file

Deployment

Insert this in the polls.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: polls
  labels:
      app: polls
spec:
  replicas: 1
  selector:
    matchLabels:
      app: polls
  template:
    metadata:
      labels:
        app: polls
    spec:
      containers:
      - name: polls-app
        image: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME/polls:${CI_COMMIT_SHORT_SHA}
        imagePullPolicy: Always
        env:
        - name: DATABASE_HOST
          value: postgresql
        - name: DATABASE_NAME
          valueFrom:
            secretKeyRef:
              name: postgresql-credentials
              key: database
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: postgresql-credentials
              key: username
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgresql-credentials
              key: password
        - name: PORT
          value: "8080"
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "100m"  
      imagePullSecrets:
        - name: registry-gitlab
      volumes:
      - name: postgresql-credentials
        secret:
          secretName: postgresql-credentials

Service

Insert this in the polls.yaml file with --- line before

apiVersion: v1
kind: Service
metadata:
  name: polls
  labels:
    app: polls
spec:
  type: ClusterIP
  ports:
  - port: 8080
  selector:
    app: polls

Ingress Resource

Insert this in the polls.yaml file with --- line before

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  labels:
    app: polls
  name: polls
spec:
  rules:
  - host: ${GITLAB_USER_LOGIN}-${CI_PROJECT_NAME}.k8s-dev.pasteur.fr
    http:
      paths:
      - backend:
          serviceName: polls
          servicePort: 8080
        path: /

Kubernetes Job

Create a file job.yaml at the root directory of your git repository and fill it with the following definition.

We will use a Job in order to manage django migrations.

Note: Kubernetes jobs are run only once opposed to Deployments that run continiously. We put it in a seperate file because a Job is immutable and cannot be updated.

Insert this in the job.yaml file

---
apiVersion: batch/v1
kind: Job
metadata:
  name: polls-migrations
spec:
  template:
      spec:
        containers:
            - name: django
              image: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME/polls:${CI_COMMIT_SHORT_SHA}
              command: ["/bin/sh","-c"]
              args: ["python manage.py makemigrations && python manage.py migrate"]
              env:
              - name: DATABASE_HOST
                value: postgresql
              - name: DATABASE_NAME
                valueFrom:
                  secretKeyRef:
                    name: postgresql-credentials
                    key: database
              - name: DATABASE_USER
                valueFrom:
                  secretKeyRef:
                    name: postgresql-credentials
                    key: username
              - name: DATABASE_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: postgresql-credentials
                    key: password
        restartPolicy: Never
        imagePullSecrets:
          - name: registry-gitlab
        volumes:
        - name: postgresql-credentials
          secret:
            secretName: postgresql-credentials
  backoffLimit: 5

Setup Continuous Delivery in Gitlab CI

Insert this in the .gitlab-ci.yml file after the previous part (docker build)

deploy:
  stage: deploy
  image: registry-gitlab.pasteur.fr/dsi-tools/docker-images:docker_kubernetes_image
  variables:
    NAMESPACE: ${GITLAB_USER_LOGIN}-${CI_PROJECT_NAME}
  environment:
    name: ${GITLAB_USER_LOGIN}-${CI_PROJECT_NAME}
    url: https://${GITLAB_USER_LOGIN}-${CI_PROJECT_NAME}.k8s-dev.pasteur.fr
  script:
    - kubectl delete secret registry-gitlab -n ${NAMESPACE} --ignore-not-found=true
    - kubectl create secret docker-registry -n ${NAMESPACE} registry-gitlab --docker-server=registry-gitlab.pasteur.fr --docker-username=${DEPLOY_USER} --docker-password=${DEPLOY_TOKEN} --docker-email=kubernetes@pasteur.fr
    - envsubst < postgresql.yaml | kubectl apply -n ${NAMESPACE} -f -
    - kubectl wait --for=condition=available --timeout=600s deployment/postgresql
    - kubectl delete job polls-migrations -n ${NAMESPACE} --ignore-not-found=true
    - envsubst < job.yaml | kubectl apply -n ${NAMESPACE} -f -
    - kubectl wait --for=condition=complete --timeout=600s job/polls-migrations
    - envsubst < polls.yaml | kubectl apply -n ${NAMESPACE} -f -
    - kubectl patch deployment polls -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
  tags:
    - k8s

Take a look at your Gitlab CI pipeline

If everything goes well, you should see that you have a pipeline with two steps:

• build
• deploy

You can see the log output by clicking on each one

Informations available in Gitlab

Gitlab offer a basic set of features in order to manage your Kubernetes web application. In the Operations section on the left panel you are able to view :

• Metrics : It will display basic memory and cpu graphs
• Environements : It will list all the environements you have created (you can have several deploy stages in your .gitlab-ci.yml : dev, stagging, production....) and give direcrt access to :
  • Website link
  • Monitoring
  • Open a remote shell on the container
• Error Tracking : If your application implement error tracking on [sentry]: https://sentry.io/ errors are displayed here.
• Serverless : Feature not available here at Pasteur

What else ?

Is my deployed application is working ?

Now we want to know if what we have done is working; to do that just go in Operations/Environements and click on the first icon on the right hand side, it should open your web app.

You may have a 404 page not found error, that's can be unfortunately normal for the first deployement, all you have to do is to restart the deploy job in the CI section of your gitlab project.

Kubernetes dashboard

We provide a web interface to visualise your workloads you have access to, you can connect to https://console.k8s-dev.pasteur.fr and follow the login process. On the left panel, you will find a drop down list and select your namespace. You will be able to view your Deployements, Pods, Services and so on.

Increase the number of replicas

If you want to increase the number of replicas of your web application, you can go in the Deployement section and select the 3 dots on the polls line; select Scale. You can increase or decrease the number of replicas. Kubernetes will automaticaly add or remove Pods.

You can also update the file polls.yaml to change the replicas value (by default 1)

Kubernetes will automatically load balance the traffic to the n PODS.

What happen if I kill a Pod ?

You can try to Delete the polls Pod using the 3 dots icon and see that Kubernetes will automaticaly restart a new one.

Grafana : Metrics and Logs

You can log on https://grafana.k8s-dev.pasteur.fr/ and browse the various dashboard available.

Best Practices

You git repository must be the source of truth, never change something directly on Kubernetes that needs to be persistent.

Integrate Kubernetes at the beginning of your project, you'll waste less time.