Gendi Notes

CI/CD

CI answers "did this change break anything?" CD answers "can this change reach users safely?" They are different problems and people conflate them constantly.

Level 2 guide ci-cdautomationpipelines

What is it?#

Stands forQuestion it answers
CIContinuous IntegrationDoes the code still build and pass its tests?
CDContinuous DeliveryIs every passing build ready to release? (a human clicks)
CDContinuous DeploymentEvery passing build is released, automatically

Both CDs are legitimate. Most teams want continuous delivery — automated all the way to production-ready, with a deliberate decision to release.

Why it exists#

Manual releases fail in predictable ways: a step gets skipped, it works on one person's machine, nobody remembers exactly what was deployed, and rolling back means reconstructing history from memory.

A pipeline makes the process written down, repeatable and identical every time. The point is not speed — it is that the same steps run in the same order whether you are calm on a Tuesday or panicking on a Friday.

What a pipeline is made of#

push / pull request │ ↓ ① BUILD compile, build the image fail fast, cheapest first │ ↓ ② TEST unit → integration → lint │ ↓ ③ PACKAGE tag the artifact with the Git SHA │ ↓ ④ PUBLISH push to a registry │ ↓ ⑤ DEPLOY staging automatic │ ↓ ⑥ VERIFY smoke tests against staging │ ↓ ⑦ DEPLOY production automatic (deployment) or gated (delivery)
StageShould beWhy
BuildFast, deterministicRuns on every push
TestOrdered cheapest-firstFail in 30s, not 30 minutes
PackageOnceBuild one artifact, promote it everywhere
DeployRepeatable, reversibleYou will need to undo it

The tool landscape#

Every tool here runs the same shape — trigger, stages, artefact, deploy. They differ in who operates them and which direction the deploy goes.

CI TOOLS (build and test) CD TOOLS (get it running) ───────────────────────── ────────────────────────── GitHub Actions Argo CD ┐ GitLab CI Flux ├ PULL: agent in the Jenkins ┘ cluster reads Git CircleCI · Buildkite Azure Pipelines the CI tool itself ┐ PUSH: pipeline Tekton Spinnaker ┘ holds cluster credentials Many tools do both. The PUSH/PULL split is the decision that matters.
ToolHosted byConfigStrongest whenWatch out for
GitHub ActionsGitHub.github/workflows/You are already on GitHubThird-party actions are a supply-chain risk — pin to SHAs
GitLab CIGitLab or you.gitlab-ci.ymlYou are on GitLab; it is deeply integratedTies you to GitLab
JenkinsYouJenkinsfileSelf-hosting is required; huge plugin ecosystemYou operate it — plugins, disk, CVEs
CircleCI / BuildkiteThem (Buildkite: your agents)YAMLFast, good caching; Buildkite keeps code on your infraPer-minute cost
Azure PipelinesMicrosoftYAMLAzure and enterprise Windows estatesAzure-centric
TektonYou, on KubernetesCRDsYou want CI as Kubernetes objectsLow-level; usually needs a UI on top
Argo CD / FluxIn your clusterGit repoKubernetes CD done properlyCD only — you still need CI

How to use it — a real GitHub Actions pipeline#

.github/workflows/deploy.yml
name: Build and deploy

on:
  push:
    branches: [main]
  pull_request:            # run tests on PRs, but do not deploy

# Cancel an in-progress run when a newer commit is pushed to the same branch.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm            # cache dependencies between runs

      # `npm ci` respects the lockfile exactly. `npm install` may resolve
      # different versions, which defeats the point of testing.
      - run: npm ci
      - run: npm run lint
      - run: npm test

  build-and-push:
    needs: test                 # only if tests passed
    if: github.ref == 'refs/heads/main'    # not on pull requests
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/build-push-action@v6
        with:
          push: true
          # Tag with the commit SHA: immutable, traceable, never ambiguous.
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    needs: build-and-push
    runs-on: ubuntu-latest
    environment: production     # attach required reviewers here for delivery
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to Kubernetes
        run: |
          kubectl set image deploy/web \
            web=ghcr.io/${{ github.repository }}:${{ github.sha }}
          # Waits for the rollout and FAILS the job if it does not succeed —
          # without this, the pipeline goes green on a broken deploy.
          kubectl rollout status deploy/web --timeout=5m

The details that matter in that file#

LineWhy it is there
concurrency + cancel-in-progressStops five queued runs when you push five times
cache: npmDependency install goes from minutes to seconds
npm ci, not npm installInstalls the lockfile exactly
needs: testNothing is published unless tests pass
if: github.ref == ...mainPRs are tested but never deployed
Tag = github.shaImmutable and traceable to a commit
environment: productionWhere you attach manual approval
rollout status --timeoutFails the job if the deploy fails

Secrets#

env:
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
RuleWhy
Never commit secretsGit history is forever, and the remote already has it
Use the platform's secret storeEncrypted, masked in logs, access-controlled
Scope by environmentStaging credentials must not reach production
Prefer short-lived tokens (OIDC)Nothing long-lived to steal
Rotate after any exposureDeleting the commit does not help

Deployment strategies#

StrategyHow it worksCost
RecreateStop old, start newDowntime. Simple
RollingReplace a few at a timeTwo versions live at once
Blue/greenFull second environment, switch trafficDouble infrastructure; instant rollback
Canary5% of traffic first, then growBest risk control; needs good metrics

Rolling is the Kubernetes default and the right starting point. Canary is worth it once you have metrics good enough to detect the problem in that 5%.

What makes a pipeline good#

FAST under 10 minutes, or people stop waiting for it RELIABLE a red build means broken code, never flakiness REVERSIBLE rollback is one command and is practised VISIBLE the log says what failed, not just that it failed

Interview corner · الأسئلة اللي بتتسأل#

A deploy needs a database migration that renames a column. Design it so there is no downtime.

The answer is that you cannot rename it in one step — during a rolling update both versions are live against the same database, so the old code would query a column that no longer exists.

You do it as an expand / contract sequence, over several deploys:

deploy 1 ADD full_name (nullable) code writes BOTH columns, reads `name` │ old pods still work: `name` is intact ↓ deploy 2 code reads `full_name`, still writes both │ backfill existing rows here ↓ deploy 3 code stops writing `name` │ nothing reads it any more ↓ deploy 4 DROP `name`

Each step is independently safe and independently rollback-able, which is the actual requirement.

Safe in one stepNeeds expand/contract
Add a nullable columnRename a column
Add a table or indexDrop a column
Widen a type (int → bigint)Narrow a type
Add a defaultAdd a NOT NULL without a default
Your CI has a step that deploys to production. Someone opens a pull request from a fork. What could go wrong?

Secrets exposure and code execution with your credentials — this is one of the most exploited CI weaknesses.

Two specific dangers:

TriggerDanger
pull_request_targetRuns in the base repo's context, with secrets, but checks out untrusted code if you are careless
A workflow that checks out the PR head and has secretsThe PR author's code runs with your tokens

An attacker opens a PR that adds one line to a build script, your pipeline runs it with secrets.AWS_ACCESS_KEY, and it posts the key to their server. The build even goes green.

Defences, in order:

# 1. Least privilege by default, per workflow
permissions:
  contents: read        # not write, and no packages/id-token unless needed

# 2. Pin third-party actions to a commit SHA, not a moving tag.
#    A tag can be repointed at new code by whoever owns the action.
- uses: actions/checkout@8f4b7f8  # v4.1.1

# 3. Separate the pipelines: test on PRs (no secrets),
#    deploy only from a push to main.
on:
  pull_request:          # test job only
  push:
    branches: [main]     # the job with deploy credentials

Plus environment: protection rules with required reviewers, and OIDC federation so there is no long-lived cloud key to steal at all.

Deploys take 40 minutes. Make them under 10 without reducing safety.

Measure the stages first — then attack in the order the time actually is:

SuspectTypical fixTypical saving
Tests run seriallyShard across parallel runnersOften 50–70%
Dependencies re-downloadedCache keyed on the lockfile hashMinutes per run
Docker layers rebuiltRegistry-backed layer cache, ordered DockerfileMinutes
Whole monorepo rebuildsBuild only changed packages from the dep graphLarge
Sequential deploy per environmentPromote the same artifact, do not rebuild1 build instead of 3
Slow tests run firstReorder cheapest-first, fail fastPerceived, but real

Common problems#

SymptomCauseFix
Passes locally, fails in CIDifferent versions, or uncommitted filesPin versions; check .gitignore
Slow pipelineNo caching; slow tests run firstCache dependencies; reorder
Deploy "succeeds", app is brokenNot waiting for the rolloutrollout status --timeout
Secret leaked in logsEchoed, or in an error messageRotate it; use the secret store
Two runs deploy at onceNo concurrency controlconcurrency: group
Cannot tell what is deployedMutable tagsTag with the Git SHA

Key takeaways#

  • CI = does it work. CD = can it ship. Different problems.
  • Build once, promote the same artifact. Rebuilding per environment invalidates your testing.
  • Tag with the Git SHA. Immutable and traceable.
  • Always wait for and verify the deploy — otherwise green means nothing.
  • Rolling updates run two versions at once, so migrations must be backwards-compatible.
  • Fix or delete flaky tests. They teach people to ignore red builds.
  • Rollback must be one command, and you must have practised it.
  • Untrusted code and production credentials never share a job.

Last updated 2026-09-08