courses

Continuous Integration and Deployment

Continuous Integration and Continuous Deployment (CI/CD) is the practice of automating the steps between writing code and running it in production. Rather than manually building, testing, and deploying software, a CI/CD pipeline runs those steps automatically every time code changes — catching bugs earlier, making releases predictable, and eliminating the “works on my machine” class of problems.

Conceptual Overview

Continuous Integration (CI)

CI is the practice of merging developer changes into a shared repository frequently — ideally several times a day — and running automated tests on every merge. The goal is to detect integration problems early, when they are cheap to fix, rather than at release time when they are expensive.

A CI system watches a repository for new commits or pull requests, then automatically:

  1. Checks out the code
  2. Builds it (compiles, installs dependencies)
  3. Runs the test suite
  4. Reports pass/fail back to the developer

Without CI: A team of five developers works for two weeks on separate branches. When they merge, the conflicts and broken assumptions between their changes require days to untangle. Bugs introduced early in the sprint are only discovered at the end.

With CI: Every push to any branch triggers a build and test run within minutes. The developer who broke something gets notified immediately, while the change is still fresh in their memory.

Continuous Delivery (CD)

Continuous Delivery extends CI by also automating the release process. After tests pass, the pipeline automatically packages the application and deploys it to a staging environment. The artifact is always in a deployable state; a human pushes a button to send it to production.

Continuous Deployment

Continuous Deployment goes one step further: every change that passes automated tests is automatically deployed to production with no human intervention. This requires a mature, high-coverage test suite and strong monitoring. It is common in software-as-a-service companies (Flickr famously deployed to production ~10 times per day in 2009; large SaaS companies now do it thousands of times per day).

The Pipeline Model

CI/CD systems model work as a pipeline — a sequence of stages, each of which must pass before the next begins:

commit → build → test → security scan → package → deploy staging → deploy production

Each stage contains one or more jobs. Jobs within a stage can run in parallel; stages run sequentially. A failure at any stage stops the pipeline and notifies the team.

Stage Typical jobs Failure means
Build Compile, install deps, lint Code won’t compile or has style errors
Test Unit tests, integration tests, coverage Logic is broken
Security SAST, dependency scan, secret detection Vulnerabilities or leaked credentials
Package Docker build, artifact creation Packaging configuration is wrong
Deploy Push to staging, smoke tests Deployment config is broken
Release Push to production, notify (Only reached if all prior stages pass)

Common Tools

GitHub Actions

GitHub’s built-in CI/CD system. Workflows are YAML files stored in .github/workflows/. Each workflow triggers on GitHub events (push, pull request, tag, schedule) and runs on runners — ephemeral virtual machines hosted by GitHub or self-hosted.

Strengths: Zero setup for GitHub-hosted repos, generous free tier (2,000 minutes/month for public repos), huge marketplace of pre-built actions, tight integration with GitHub pull requests and branch protections.

Weaknesses: Tied to GitHub; complex pipelines can become unwieldy YAML.

GitLab CI/CD

Built into GitLab, configured via .gitlab-ci.yml at the repo root. The strongest choice when the entire DevOps toolchain (issue tracking, CI, security scanning, container registry, deployment) should live in one platform.

Strengths: All-in-one platform, strong built-in security scanning (SAST, DAST, dependency scanning, secret detection), native Kubernetes integration, self-hosted option with full control.

Weaknesses: More complex to self-host; YAML syntax has more concepts to learn.

Jenkins

Self-hosted, open-source automation server. Pipelines are defined in a Jenkinsfile (Groovy DSL). The oldest and most flexible option — over 1,800 plugins cover virtually any integration.

Strengths: Runs anywhere (on-premises, air-gapped), extreme customization, massive plugin ecosystem, used in most Fortune 500 environments.

Weaknesses: Significant operational overhead (you maintain the server), Groovy syntax has a learning curve, UI is dated.

Others

Tool Best for
CircleCI Fast builds with good Docker support
Argo CD GitOps-style continuous deployment to Kubernetes
Flux GitOps for Kubernetes, declarative approach
Tekton Kubernetes-native pipeline primitives
Drone Lightweight, container-first, self-hosted

Rule of thumb (2026): If your code is on GitHub, start with GitHub Actions. If you need a full self-hosted DevOps platform, use GitLab. If you’re in an enterprise with complex legacy integrations, Jenkins is likely already there.

GitHub Actions

Workflow file structure

Workflows live in .github/workflows/<name>.yml. A minimal example:

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest --tb=short

Key concepts:

Triggers

on:
  push:                        # any push to any branch
  push:
    branches: [main, develop]  # only these branches
    tags: ['v*']               # or version tags
  pull_request:
    types: [opened, synchronize]
  schedule:
    - cron: '0 3 * * 1'        # every Monday at 03:00 UTC
  workflow_dispatch:           # manual trigger from the GitHub UI

Matrix builds

Run the same job across multiple environments simultaneously:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        os: [ubuntu-latest, windows-latest]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: $
      - run: pytest

This creates 6 parallel jobs (3 Python versions × 2 OSes).

Secrets and environment variables

Never put passwords, API keys, or tokens in workflow files. Store them as GitHub Secrets (Settings → Secrets and variables → Actions) and reference them with $:

steps:
  - name: Deploy to server
    env:
      DEPLOY_KEY: $
      SERVER: $   # non-secret config uses vars
    run: |
      echo "$DEPLOY_KEY" | ssh-add -
      rsync -av dist/ user@$SERVER:/var/www/app/

Secrets are masked in logs — if a step accidentally prints one, GitHub replaces it with ***.

Artifacts and caching

steps:
  # Cache pip dependencies between runs
  - uses: actions/cache@v4
    with:
      path: ~/.cache/pip
      key: $-pip-$

  - run: pip install -r requirements.txt

  - run: pytest --junitxml=results.xml

  # Upload test results for later download
  - uses: actions/upload-artifact@v4
    if: always()   # upload even if tests fail
    with:
      name: test-results
      path: results.xml

Worked example: Jekyll site deployment

This is the pipeline structure used for deploying a Jekyll site (like this one) to a web server on every push to main:

name: Build and Deploy

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.3'
          bundler-cache: true   # runs bundle install and caches gems

      - name: Build site
        run: bundle exec jekyll build
        env:
          JEKYLL_ENV: production

      - name: Upload built site
        uses: actions/upload-artifact@v4
        with:
          name: site
          path: _site/

  deploy:
    needs: build          # only runs if build job passes
    runs-on: ubuntu-latest
    environment: production   # requires approval if configured

    steps:
      - uses: actions/download-artifact@v4
        with:
          name: site
          path: _site/

      - name: Deploy via rsync
        env:
          SSH_KEY: $
          DEPLOY_HOST: $
          DEPLOY_PATH: $
        run: |
          echo "$SSH_KEY" > /tmp/deploy_key
          chmod 600 /tmp/deploy_key
          rsync -az --delete \
            -e "ssh -i /tmp/deploy_key -o StrictHostKeyChecking=no" \
            _site/ deploy@$DEPLOY_HOST:$DEPLOY_PATH

Worked example: Docker build and push

name: Build and Push Container

on:
  push:
    tags: ['v*.*.*']

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: $
          password: $   # automatically provided

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/$:$
          cache-from: type=gha
          cache-to: type=gha,mode=max

GitLab CI/CD

Pipeline file structure

Everything lives in .gitlab-ci.yml at the repository root.

stages:
  - build
  - test
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip"

build-job:
  stage: build
  image: python:3.12
  cache:
    paths:
      - .pip/
  script:
    - pip install -r requirements.txt
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

test-job:
  stage: test
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - pytest --junitxml=report.xml
  artifacts:
    reports:
      junit: report.xml   # GitLab parses this and shows test results in the MR
    when: always

deploy-staging:
  stage: deploy
  script:
    - rsync -az dist/ user@staging.example.com:/var/www/app/
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - main

Key concepts:

Rules (modern condition syntax)

rules: replaces only: / except: and is more expressive:

deploy-production:
  stage: deploy
  script:
    - ./deploy.sh production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/   # version tags only
      when: manual                                  # requires human approval
    - when: never

Environments and deployments

GitLab tracks deployments to named environments and shows deployment history:

deploy-prod:
  stage: deploy
  environment:
    name: production
    url: https://app.example.com
    on_stop: stop-prod   # job to run when environment is "stopped"
  script:
    - kubectl apply -f k8s/

stop-prod:
  stage: deploy
  environment:
    name: production
    action: stop
  script:
    - kubectl delete -f k8s/
  when: manual

Built-in security scanning

GitLab CI has first-class support for security stages via pre-built templates:

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml

stages:
  - test
  - security
  - deploy

Including these templates adds scanning jobs automatically. Results appear in the merge request as a security report — blocking merge if critical findings are present (configurable).

Worked example: multi-stage Python application

image: python:3.12-slim

stages:
  - lint
  - test
  - build
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"

.python-base: &python-base
  cache:
    key: python-$CI_COMMIT_REF_SLUG
    paths: [.pip-cache/]
  before_script:
    - pip install --cache-dir .pip-cache -r requirements.txt

lint:
  <<: *python-base
  stage: lint
  script:
    - pip install ruff
    - ruff check .
    - ruff format --check .

unit-tests:
  <<: *python-base
  stage: test
  script:
    - pytest tests/unit/ --cov=app --cov-report=xml --junitxml=junit.xml
  coverage: '/TOTAL.*\s(\d+%)$/'   # extracts coverage % for display in GitLab UI
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

integration-tests:
  <<: *python-base
  stage: test
  services:
    - postgres:16        # spins up a Postgres container alongside the job
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test
    DATABASE_URL: postgresql://test:test@postgres/testdb
  script:
    - pytest tests/integration/

build-image:
  stage: build
  image: docker:27
  services:
    - docker:27-dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-staging:
  stage: deploy
  script:
    - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  environment:
    name: staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Jenkins

Jenkins pipelines are defined in a Jenkinsfile using a Groovy-based DSL. There are two syntaxes: Declarative (structured, recommended) and Scripted (more flexible but verbose).

Declarative pipeline example

pipeline {
    agent any   // run on any available agent

    environment {
        DEPLOY_SERVER = 'deploy@app.example.com'
    }

    stages {
        stage('Build') {
            steps {
                sh 'mvn -B package -DskipTests'
            }
        }

        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/*.xml'
                }
            }
        }

        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                withCredentials([sshUserPrivateKey(
                    credentialsId: 'deploy-key',
                    keyFileVariable: 'SSH_KEY'
                )]) {
                    sh 'rsync -az -e "ssh -i $SSH_KEY" target/app.jar $DEPLOY_SERVER:/apps/'
                }
            }
        }
    }

    post {
        failure {
            mail to: 'team@example.com',
                 subject: "Build failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                 body: "See ${env.BUILD_URL}"
        }
    }
}

Jenkins is most commonly chosen when:

Key Concepts Across All Tools

Branch strategies

CI/CD behavior typically varies by branch:

Branch CI behavior CD behavior
Feature branches Run lint + unit tests No deployment
main / develop Full test suite Deploy to staging automatically
Release tags (v1.2.3) Full test suite + security scan Deploy to production (manually or automatically)

Fail fast

Order pipeline stages so cheap checks (linting, type checking) run before expensive ones (integration tests, Docker builds). A 30-second lint failure should not make developers wait 10 minutes for a test suite to finish before getting feedback.

Idempotent deployments

Deployment scripts should produce the same result whether run once or ten times. Use declarative tools (Kubernetes manifests, Ansible, Terraform) rather than imperative scripts that assume a particular starting state.

Secrets management

Never commit secrets to the repository — not even to private repos. CI/CD tools provide secrets stores:

Platform Secrets mechanism
GitHub Actions Repository / organization secrets ($)
GitLab CI CI/CD variables (masked, protected)
Jenkins Credentials store (withCredentials())
All External: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault

Use short-lived credentials where possible. OIDC (OpenID Connect) federation allows GitHub Actions and GitLab CI to authenticate to cloud providers (AWS, GCP, Azure) without storing long-lived keys — the CI system presents a JWT token that the cloud provider verifies directly.

Pipeline as code

The pipeline definition lives in the repository alongside the application code. This means:

Further Reading