Three commands solve most problems: `describe` tells you what Kubernetes thinks, `logs` tells you what the app said, `get endpoints` tells you why traffic is not arriving.
Whatever the symptom, this order wastes the least time:
something is wrong
│
↓
① kubectl get pods what state is it in?
│
↓
② kubectl describe pod <p> read the EVENTS at the bottom.
│ They usually name the problem outright.
↓
③ kubectl logs <p> --previous what did the app say before it died?
│
↓
④ kubectl get endpoints <svc> is traffic even reaching it?
The four commands
kubectl get pods -o wide
kubectl describe pod <pod> # scroll to Events
kubectl logs <pod> --previous # --previous = the crashed container
kubectl get endpoints <service>
This is not an error type. It means "the container keeps exiting, so I am restarting it more slowly each time" — the delay grows 10s, 20s, 40s… capped at 5 minutes. It tells you nothing about the cause.
The exit code does.
kubectl describe pod <pod> | grep -A6 "Last State"
Exit code
Meaning
Where to look
0
Exited successfully — the command finished
Wrong command; use a Job for one-shot work
1 / 2
Application error
kubectl logs --previous
137
128+9 = SIGKILL — usually OOM
Memory limit
139
128+11 = segfault
Often a wrong-architecture image (arm64 vs amd64)
143
128+15 = SIGTERM
Something asked it to stop — often a liveness probe
Then the usual suspects:
kubectl logs <pod> --previous # config errors, missing env vars, stack traces
kubectl describe pod <pod> | grep -i probe # is a probe killing it?
kubectl get configmap,secret # does everything it references exist?
Debugging a container that dies too fast to exec into#
# Copy the pod with the entrypoint replaced by a shell. The copy does NOT
# crash, so you can inspect the config, env vars and mounts it really sees.
kubectl debug <pod> -it --copy-to=debug --container=<name> -- sh
# Distroless image with no shell? Attach a container that shares its namespaces.
kubectl debug -it <pod> --image=busybox --target=<container>
kubectl describe pod <pod> | grep -i -A3 "last state"
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}{"\n"}'
Either the limit is too low, or the app leaks. Two frequent traps:
A runtime sizing itself from the host. Older JVMs and some Node configurations read the machine's total memory and size their heap to it, then get killed for exceeding a much smaller container limit. Modern JVMs are container-aware; older ones need -XX:MaxRAMPercentage set explicitly.
A monitoring graph showing headroom. Metrics are sampled every 15–60 seconds; the spike that triggered the kill lasted milliseconds. kubectl describe reports the kernel's account, which is authoritative.
The pod is up. Traffic does not arrive, or the app misbehaves.
# 1. Does the Service have backends? This is the answer most of the time.
kubectl get endpoints <service>
# 2. Is the pod actually Ready? 0/1 means the readiness probe is failing,
# so the Service is correctly refusing to route to it.
kubectl get pods
# 3. Bypass the Service — talk straight to a pod IP.
# If this works, your problem is the Service or DNS, not the app.
kubectl get pod <pod> -o wide # note the IP
kubectl run tmp --rm -it --image=busybox --restart=Never -- \
wget -qO- --timeout=3 http://<pod-ip>:8080/healthz
# 4. Is DNS working inside the cluster?
kubectl run tmp --rm -it --image=busybox --restart=Never -- nslookup <service>
# 5. Is a NetworkPolicy blocking it? Policies apply to the DESTINATION.
kubectl get networkpolicy -A
kubectl get pod <pod> -o yaml | grep -A5 finalizers
Two causes:
The app ignores SIGTERM, so it waits out terminationGracePeriodSeconds (default 30) and is then killed. The tell is that termination always takes exactly the same number of seconds.
A finalizer is waiting on something that will never complete — a volume detach, an external controller that has been removed.
kubectl delete pod <pod> --grace-period=0 --force # last resort
kubectl get nodes # all Ready?
kubectl describe node <node> | grep -A10 "Allocated resources"
kubectl top nodes # actual usage (needs metrics-server)
kubectl top pods -A --sort-by=memory
# Events across the whole cluster, newest last. Often shows the real story.
kubectl get events -A --sort-by=.lastTimestamp | tail -30
# Is the control plane healthy?
kubectl get pods -n kube-system
kubectl get events -A --sort-by=.lastTimestamp is underused. When several things break at once it usually reveals the single underlying cause.
A pod is in CrashLoopBackOff. Take me through your process.
The trap is answering with a cause. CrashLoopBackOff is a restart policy, not a diagnosis — it means "this keeps exiting so I am backing off" and carries no information about why.
# 1. Get the exit code. This is the actual diagnostic signal.
kubectl describe pod <pod> | grep -A6 "Last State"
# 2. Read the DEAD container's logs, not the new one's.
kubectl logs <pod> --previous
# 3. If it dies too fast to inspect, copy it with the entrypoint replaced.
kubectl debug <pod> -it --copy-to=dbg --container=app -- sh
Then the exit code tells you which direction to go:
Code
Where the fault is
0
Not a crash. The command completed. Wrong command, or it should be a Job
1/2
Application. Logs will have a stack trace or config error
137
Memory limit (SIGKILL/OOM), or an unhandled SIGTERM timing out
139
Segfault — very often an amd64 image on arm64 hardware
143
Something asked it to stop. Usually the liveness probe
Pods are Pending but kubectl top nodes shows the cluster is only 30% used. Explain.
Because the scheduler does not look at usage at all. It looks at requests — capacity you have already reserved whether you use it or not.
Node: 8 CPU total
─────────────────────────────────────────────
Actual usage (what `top` shows): ██░░░░░░ 2 CPU = 25%
Sum of requests (what the
scheduler counts): ███████░ 7 CPU = 87% reserved
Your pod requests 2 CPU → 8 − 7 = 1 available → Pending
# The number that actually matters — reserved, not used
kubectl describe node <node> | grep -A8 "Allocated resources"
The fix is almost never "add nodes". It is to right-size requests: measure real consumption with kubectl top pods, then set requests near that plus a margin. Inflated "just to be safe" requests are how clusters end up 80% reserved and 25% used.
A pod has been Terminating for 20 minutes. What is holding it, and is --force safe?
Two causes, and they need different responses.
1. The app ignores SIGTERM. It waits out terminationGracePeriodSeconds (30 by default) and is then SIGKILLed. The tell is that deletion always takes exactly the same number of seconds. Fix the app, not the deletion.
2. A finalizer. A finalizer is a "do not delete until I say so" marker. The object stays until whatever owns that finalizer removes it — and if that controller has been uninstalled, it never will.
kubectl get pod <pod> -o yaml | grep -A5 finalizers
kubectl describe pod <pod> | tail -20 # volume detach errors?
Is --force safe? For a stateless pod, yes. For anything with state, no.
kubectl delete pod <pod> --grace-period=0 --force
It deletes the API object without waiting for the node to confirm the container is gone. With a StatefulSet or an attached ReadWriteOnce volume, Kubernetes will now happily start a replacement while the original process may still be running and holding that disk — two writers on one filesystem, which is how you corrupt a database.