Gendi Notes

Jenkins

Jenkins is a job runner with an enormous plugin ecosystem — which is both why it survives everywhere and why every neglected Jenkins becomes a liability.

Level 2 guide jenkinsci-cdautomation

What is it?#

Jenkins runs automation jobs on a schedule or a trigger. A controller schedules work and stores configuration; agents execute it. The job itself is a Jenkinsfile committed alongside your code.

git push / webhook / cron / manual │ ↓ ┌──────────────────┐ assigns work │ CONTROLLER │ ─────────────────────┐ │ · schedules │ ↓ │ · stores config │ ┌──────────────────┐ │ · serves the UI │ │ AGENT (node) │ │ · NEVER builds │ ←──────────│ runs the stages │ └──────────────────┘ results │ has the tools │ └──────────────────┘ static VM · Docker · Kubernetes pod

Why it still exists#

It is old, its UI shows it, and it is still everywhere. The honest reasons:

ReasonWhat it means in practice
It runs anywhereYour own hardware, air-gapped networks, regulated environments
~1,800 pluginsThere is an integration for essentially anything
No per-minute billingAt high build volume, self-hosted is dramatically cheaper
It predates the alternativesFifteen years of accumulated pipelines nobody will rewrite

And the honest costs:

CostWhy
You operate itUpgrades, disk, backups, plugin CVEs — it is a service you own
Plugins are the attack surfaceMost Jenkins CVEs are plugin CVEs
Groovy is a real languageShared libraries become software that itself needs tests
Config drifts from codeUnless you enforce Configuration as Code

What it is made of#

PieceWhat it is
ControllerSchedules, stores config in $JENKINS_HOME, serves the UI
Agent / nodeWhere builds actually run
ExecutorOne concurrent build slot on an agent
JobA thing that can run. Usually a Pipeline
JenkinsfileThe pipeline, as code, in your repo
StageA named phase, shown as a column in the UI
Shared libraryGroovy code reused across pipelines
CredentialsSecrets, injected into builds and masked in logs

How to use it#

A declarative pipeline#

Jenkinsfile — declarative, which is what you want
pipeline {
  // Run each stage in a fresh container. Nothing persists between builds,
  // which is the whole point — no "works on agent-3" mysteries.
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
            - name: node
              image: node:22-alpine
              command: ['cat']
              tty: true
      '''
    }
  }

  options {
    timeout(time: 30, unit: 'MINUTES')     // never let a job hang forever
    disableConcurrentBuilds()              // one build per branch at a time
    buildDiscarder(logRotator(numToKeepStr: '30'))   // or the disk fills
  }

  environment {
    IMAGE = "registry.example.com/web"
    // Never interpolate a credential with ${} — see the warning below
    REGISTRY = credentials('registry-creds')
  }

  stages {
    stage('Test') {
      steps {
        container('node') {
          sh 'npm ci'
          sh 'npm test -- --ci --reporters=jest-junit'
        }
      }
      post {
        always { junit 'junit.xml' }       // publish results even on failure
      }
    }

    stage('Build and push') {
      when { branch 'main' }               // PRs are tested, never published
      steps {
        container('node') {
          // Tag with the commit SHA: immutable and traceable
          sh 'docker build -t $IMAGE:$GIT_COMMIT .'
          sh 'docker push $IMAGE:$GIT_COMMIT'
        }
      }
    }

    stage('Deploy') {
      when { branch 'main' }
      steps {
        // Wait for the rollout, and FAIL if it does not succeed
        sh 'kubectl set image deploy/web web=$IMAGE:$GIT_COMMIT'
        sh 'kubectl rollout status deploy/web --timeout=5m'
      }
    }
  }

  post {
    failure { slackSend(channel: '#alerts', message: "Build ${env.BUILD_URL} failed") }
    always  { cleanWs() }
  }
}

Credentials#

withCredentials([usernamePassword(
    credentialsId: 'registry-creds',
    usernameVariable: 'USER',
    passwordVariable: 'PASS')]) {
  // Single quotes: the shell expands $PASS, Groovy never sees the value
  sh 'echo "$PASS" | docker login -u "$USER" --password-stdin registry.example.com'
}

Multibranch and webhooks#

A Multibranch Pipeline scans a repository and creates a job per branch and per pull request automatically, each running that branch's own Jenkinsfile.

repo ├── main → job, runs Jenkinsfile from main ├── feature/x → job, created automatically on push └── PR #42 → job, runs the PR's Jenkinsfile (from a fork? see the security warning)

Trigger it with a webhook rather than polling — polling every minute across 200 repos is wasted load and slow feedback.

Configuration as Code#

A Jenkins configured by clicking is unreproducible. JCasC puts it in YAML:

jenkins.yaml
jenkins:
  systemMessage: "Managed by JCasC. Do not configure through the UI."
  numExecutors: 0            # the controller NEVER builds. See below.
  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: "admin"
            permissions: ["Overall/Administer"]
            assignments: ["platform-team"]
  clouds:
    - kubernetes:
        name: "k8s"
        namespace: "jenkins"
        jenkinsUrl: "http://jenkins.jenkins.svc:8080"

unclassified:
  location:
    url: "https://jenkins.example.com/"

What goes wrong#

SymptomCauseFix
Disk full on the controllerBuild history and artefacts accumulatebuildDiscarder in every job
Builds queue but never startNo agent matches the agent labelCheck agent labels and connectivity
java.lang.OutOfMemoryErrorController heap too small for the job countRaise heap; move work to agents
Works on one agent, not anotherAgents are not identical — the classic pet problemEphemeral containerised agents
Plugin upgrade breaks everythingPlugins depend on core and each otherStaging Jenkins; pin versions; back up $JENKINS_HOME first
Credential visible in the logDouble-quoted sh stepSingle quotes; rotate the credential
Pipeline passes, deploy is brokenNo rollout statusWait for and verify the deploy
Scripts not permitted to use method...Groovy sandbox blocked a callApprove in In-process Script Approval, or avoid the call

Interview corner · الأسئلة اللي بتتسأل#

Why should the Jenkins controller have zero executors?

Because a build runs code from a repository, and the controller holds every secret you have.

$JENKINS_HOME contains credentials, job definitions, plugin configuration and the private key that identifies the instance. If builds execute there, any repository Jenkins builds can read all of it — and pull requests from forks can modify the Jenkinsfile that does the reading.

Setting numExecutors: 0 and running all work on agents means a compromised build gets an ephemeral container, not your CI system.

Declarative or scripted pipelines, and why?

Declarative, and the reason is that its constraints are the feature.

DeclarativeScripted
Validated before the build startsYesNo — fails mid-run
Renders correctly in the UIYesPartially
Expressive powerDeliberately limitedFull Groovy
Long-term outcomeA readable pipelineA bespoke program only its author understands

Scripted pipelines let each team invent its own control flow, and over a few years you get Groovy applications with no tests sitting in the deploy path for production. Declarative keeps pipelines comparable across repositories, which is what lets a platform team support them at all.

For the genuinely dynamic case, declarative allows a script { } block — so you keep the structure and escape only where you must.

Jenkins versus GitHub Actions versus GitLab CI — how do you choose?

Ask what constraint you actually have, then pick:

JenkinsGitHub ActionsGitLab CI
Who operates itYouGitHubGitLab (or you)
Runs air-gappedYesSelf-hosted runners onlyYes
Cost modelYour hardwarePer-minutePer-minute or self-hosted
Config lives inJenkinsfile.github/workflows/.gitlab-ci.yml
Ecosystem~1,800 pluginsMarketplace actionsBuilt-in, fewer add-ons
Main riskPlugin CVEs, neglectSupply chain: third-party actionsCoupling to GitLab

Decision rules:

  • Must self-host for regulatory or network reasons → Jenkins.
  • Already on GitHub, normal project → Actions. Do not run CI you could rent.
  • Already on GitLab → GitLab CI; the integration is the point.
  • Very high build volume where per-minute billing hurts → self-hosted runners for the hosted product, before rebuilding on Jenkins.

Key takeaways#

  • Controller schedules, agents build. numExecutors: 0 on the controller is a security control, not a tuning knob.
  • Declarative pipelines, in a Jenkinsfile committed with the code.
  • Single quotes for any sh touching a credential — double quotes let Groovy interpolate the secret into the log.
  • buildDiscarder and timeout in every job, or the disk fills and jobs hang forever holding executors.
  • Wait for the deploy. rollout status, or the pipeline lies.
  • Fork pull requests are untrusted code. Never build them with credentials.
  • Configuration as Code, or your Jenkins is unreproducible.
  • Plugins are the attack surface — update quarterly on staging, and test your $JENKINS_HOME restore.

Last updated 2026-09-13