Skip to content
Snippets Groups Projects
  • Régis Behmo's avatar
    bce6432d
    Improve job running in local and k8s · bce6432d
    Régis Behmo authored
    Running jobs was previously done with "exec". This was because it
    allowed us to avoid copying too much container specification information
    from the docker-compose/deployments files to the jobs files. However,
    this was limiting:
    
    - In order to run a job, the corresponding container had to be running.
    This was particularly painful in Kubernetes, where containers are
    crashing as long as migrations are not correctly run.
    - Containers in which we need to run jobs needed to be present in the
    docker-compose/deployments files. This is unnecessary, for example when
    mysql is disabled, or in the case of the certbot container.
    
    Now, we create dedicated jobs files, both for local and k8s deployment.
    This introduces a little redundancy, but not too much. Note that
    dependent containers are not listed in the docker-compose.jobs.yml file,
    so an actual platform is still supposed to be running when we launch the
    jobs.
    
    This also introduces a subtle change: now, jobs go through the container
    entrypoint prior to running. This is probably a good thing, as it will
    avoid forgetting about incorrect environment variables.
    
    In k8s, we find ourselves interacting way too much with the kubectl
    utility. Parsing output from the CLI is a pain. So we need to switch to
    the native kubernetes client library.
    bce6432d
    History
    Improve job running in local and k8s
    Régis Behmo authored
    Running jobs was previously done with "exec". This was because it
    allowed us to avoid copying too much container specification information
    from the docker-compose/deployments files to the jobs files. However,
    this was limiting:
    
    - In order to run a job, the corresponding container had to be running.
    This was particularly painful in Kubernetes, where containers are
    crashing as long as migrations are not correctly run.
    - Containers in which we need to run jobs needed to be present in the
    docker-compose/deployments files. This is unnecessary, for example when
    mysql is disabled, or in the case of the certbot container.
    
    Now, we create dedicated jobs files, both for local and k8s deployment.
    This introduces a little redundancy, but not too much. Note that
    dependent containers are not listed in the docker-compose.jobs.yml file,
    so an actual platform is still supposed to be running when we launch the
    jobs.
    
    This also introduces a subtle change: now, jobs go through the container
    entrypoint prior to running. This is probably a good thing, as it will
    avoid forgetting about incorrect environment variables.
    
    In k8s, we find ourselves interacting way too much with the kubectl
    utility. Parsing output from the CLI is a pain. So we need to switch to
    the native kubernetes client library.