Gendi Notes

StatefulSets, DaemonSets and Jobs

A Deployment assumes its pods are interchangeable. The other three workload types exist for the cases where they are not — because they have identity, because you need one per node, or because they are meant to finish.

Level 3 guide kubernetesworkloadsstatefulset

What is it?#

Four controllers, each for a different assumption about the workload.

ControllerCreatesUse for
DeploymentN identical, disposable podsStateless apps. Most things
StatefulSetN pods with stable identity and storageDatabases, queues, anything clustered
DaemonSetOne pod per nodeLog shippers, metrics agents, CNI
Job / CronJobPods that run to completionMigrations, batch, backups
DEPLOYMENT STATEFULSET DAEMONSET JOB ────────── ─────────── ───────── ─── web-7d4f-a1b2 db-0 node-1: agent migrate-x9k web-7d4f-c3d4 db-1 node-2: agent │ web-7d4f-e5f6 db-2 node-3: agent ↓ completes, random names STABLE names follows nodes then stops any order ordered 0,1,2 automatically shared or no PVC PVC PER POD usually hostPath

StatefulSet#

What it guarantees#

DEPLOYMENT STATEFULSET ────────── ─────────── name: web-7d4f9c-a1b2 name: db-0, db-1, db-2 ← stable, ordinal dies → new random name dies → comes back as db-1 exactly all start at once starts 0, then 1, then 2 ← ordered scale down: random pod scale down: highest ordinal first one shared PVC (or none) db-0 → data-db-0 ← PVC per pod db-1 → data-db-1 no per-pod DNS db-0.mysvc.ns.svc.cluster.local
statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: db }
spec:
  serviceName: db-headless          # REQUIRED, and must be a headless Service
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: postgres
          image: postgres:17
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:             # creates ONE PVC PER POD
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources: { requests: { storage: 100Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: db-headless }
spec:
  clusterIP: None                   # headless → per-pod DNS records
  selector: { app: db }
  ports: [{ port: 5432 }]

DaemonSet#

One pod per node, automatically — including on nodes that join later.

daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata: { name: log-agent }
spec:
  selector:
    matchLabels: { app: log-agent }
  template:
    metadata:
      labels: { app: log-agent }
    spec:
      # Run on control-plane nodes too, which are usually tainted
      tolerations:
        - operator: Exists
      containers:
        - name: agent
          image: fluent-bit:3.1
          resources:
            limits: { memory: 200Mi }     # it runs EVERYWHERE — cap it
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath: { path: /var/log }
Use forWhy a DaemonSet
Log collectionMust read /var/log on every node
Metrics agentsnode-exporter measures the node itself
CNI / kube-proxyNetworking must exist on every node
Storage driversCSI node plugins mount volumes locally

Job and CronJob#

job.yaml
apiVersion: batch/v1
kind: Job
metadata: { name: migrate }
spec:
  backoffLimit: 3           # retries before marking Failed
  completions: 1            # how many successful pods are needed
  parallelism: 1            # how many may run at once
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600   # auto-delete, or Jobs accumulate forever
  template:
    spec:
      restartPolicy: OnFailure    # Never or OnFailure. NOT Always
      containers:
        - name: migrate
          image: myapp:1.4.2
          command: ["./migrate.sh"]
cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata: { name: backup }
spec:
  schedule: "0 2 * * *"           # 02:00 — in the CONTROLLER's timezone
  timeZone: "Africa/Cairo"        # 1.27+. Set it explicitly
  concurrencyPolicy: Forbid       # do not start if the last one still runs
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: backup:1.0
FieldWhy it matters
restartPolicyAlways is invalid — a Job is meant to end
backoffLimitDefault 6, with exponential backoff
ttlSecondsAfterFinishedWithout it, finished Jobs pile up until etcd suffers
concurrencyPolicyAllow (default), Forbid, Replace
timeZoneOtherwise it is the controller's zone, which is usually UTC

What goes wrong#

SymptomControllerCause
db-1 never startsStatefulSetdb-0 is not Ready — startup is ordered
PVCs remain after deleteStatefulSetBy design; delete them explicitly
DaemonSet skips some nodesDaemonSetNode taints without matching tolerations
Every node OOMs at onceDaemonSetNo memory limit on the agent
Job reruns foreverJobrestartPolicy: Always, or backoffLimit too high
CronJobs pile upCronJobNo ttlSecondsAfterFinished or history limits
Two backups at onceCronJobconcurrencyPolicy: Allow
CronJob skipped a runCronJobMissed startingDeadlineSeconds while the controller was down

Interview corner · الأسئلة اللي بتتسأل#

When would you use a StatefulSet instead of a Deployment?

When the pods are not interchangeable — when something depends on which pod it is. Three guarantees a Deployment cannot give:

GuaranteeWhy it matters
Stable network identitydb-0 is always db-0. Replicas configure "replicate from db-0"
Stable storagedb-1 always reattaches data-db-1, not someone else's data
Ordered, one-at-a-time operationsA quorum system must not restart all members at once

Concretely: PostgreSQL with replicas, Kafka, Elasticsearch, ZooKeeper, etcd — anything where members know about each other.

And the honest counter-argument, worth volunteering: a StatefulSet gives you identity and storage, but it does not make an application clustered. It will not elect a leader, replicate data, or rejoin a split brain. That logic is the application's, and for production databases an operator — or a managed service — is usually the right answer rather than a hand-written StatefulSet.

A CronJob's backups start overlapping. What happened and how do you fix it?

The job now takes longer than its schedule interval, and the default concurrencyPolicy: Allow lets a new run start while the old one is going.

schedule: hourly, runtime grew to 90 minutes 10:00 ████████████████ 11:30 11:00 ████████████████ 12:30 ← two at once 12:00 ████████████████ ← three, and accelerating │ └─ all hammering the same database

The fix is one field:

concurrencyPolicy: Forbid     # or Replace, if the newest run is the useful one

But the fix alone is not the answer — with Forbid, runs are now skipped instead of overlapping, which is quieter and can mean backups silently stop happening. So you also need:

  • Alerting on job age: if the last successful completion is older than two intervals, page someone.
  • activeDeadlineSeconds so a hung run cannot block every subsequent one indefinitely.
  • Investigate the growth. A backup that grew from 20 to 90 minutes is telling you something about data volume that will not stop growing.

Key takeaways#

  • Deployment for stateless, StatefulSet for identity, DaemonSet for per-node, Job for work that ends.
  • StatefulSets give identity and storage, not clustering. The application still has to know how to be a cluster.
  • A headless Service is required for per-pod DNS.
  • volumeClaimTemplates PVCs survive deletion — deliberately, and expensively if forgotten.
  • DaemonSets need limits and tolerations, or they OOM every node or skip the control plane.
  • restartPolicy: Always is invalid in a Job, and ttlSecondsAfterFinished stops them accumulating.
  • concurrencyPolicy: Forbid for anything touching shared state — then alert on skipped runs.

Last updated 2026-09-13