The round that decides most offers, and the one people prepare least — because it is the only round where a confident guess scores worse than an honest method.
Nine scenarios of the kind handed to you in a live debugging round, each with the investigation an interviewer wants to hear.
These are not knowledge questions. Every one is answerable by someone who knows less than you and has a method, and failable by someone who knows more and starts naming tools.
Because the failure mode is so consistent. Handed a broken system, most candidates start executing:
WHAT MOST PEOPLE DO WHAT SCORES
─────────────────── ───────────
"I'd check the logs" "First — what changed?"
"I'd look at Grafana" "How wide is it? One user or all?"
"I'd restart it" "Which layer? Let me rule out DNS."
│ │
↓ ↓
runs commands until each step eliminates a
something looks wrong class of cause, out loud
│ │
↓ ↓
finds a decoy, declares narrows to the real cause,
victory, is wrong and can say why
The interviewer is not checking whether you know kubectl describe. They are checking whether, at 3am with incomplete information, you converge or thrash.
Learn this shape and every scenario below becomes an instance of it.
1. WHAT CHANGED? deploys · config · certs · DNS · traffic · quota
│ ~70% of incidents end at this question
↓
2. HOW WIDE? one user / one pod / one node / one AZ / all
│ narrow → client, or that instance
│ wide → shared dependency, or config
↓
3. WHICH LAYER? DNS → TCP → TLS → HTTP → app → datastore
│ walk it in order. do not jump
↓
4. PROVE IT one command whose result halves the possibilities
│
↓
5. STOP THE BLEEDING roll back first, understand second
— and say this part out loud
The site returns 502 for about 1% of requests. Deploys happen four times a day. Go. · ٥٠٢ لـ ١٪ من الريكوستات
The two numbers in the question are the answer: 1% and four deploys a day. Intermittent, low-percentage errors that correlate with deploys are the endpoint de-registration race.
# 1. Do the 502s cluster in time? Correlate with deploy timestamps.
kubectl rollout history deploy/web
# 2. During a deploy, watch pods leave and arrive
kubectl get pods -l app=web -w
# 3. The mechanism: is there a preStop hook and a SIGTERM handler?
kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].lifecycle}{"\n"}'
kubectl get deploy web -o jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}{"\n"}'
The mechanism to explain: when a pod is deleted, two things happen in parallel with no ordering guarantee — the kubelet sends SIGTERM, and the endpoints controller removes the pod, after which every node must reprogram its rules. In that window the pod is shutting down while nodes still route to it.
Plus an application that drains on SIGTERM instead of exiting immediately.
Half the pods in a Deployment are Pending. kubectl top nodes shows 30% utilisation. · نص البودات Pending والكلاستر فاضي
The scheduler does not read utilisation. It reads requests — reserved capacity, used or not.
# The number that matters: reserved, not used
kubectl describe node <node> | grep -A8 "Allocated resources"
# And why THIS pod could not be placed — Events say it outright
kubectl describe pod <pending-pod> | tail -20
Read the Event message, because each one points somewhere different:
Event
Cause
Fix
Insufficient cpu / memory
Requests exceed free reserved capacity
Right-size requests; add nodes
untolerated taint
Nodes reserved for other workloads
Add a matching toleration
didn't match Pod's node affinity
Affinity excludes every node
Relax it
unbound immediate PersistentVolumeClaims
No matching PV / wrong AZ
kubectl get pvc, check zone
too many pods
Node's pod-count limit reached, not CPU/RAM
Larger nodes, or raise the kubelet limit
The last row is the one people miss: a node has a maximum pod count (110 by default, and much lower on some cloud node types by IP-per-ENI limits). A node can be idle and still full.
A pod restarts every few minutes. Logs show nothing wrong and the app team says the code is fine. · بود بيعمل restart كل شوية واللوجز نضيفة
"Logs show nothing wrong" plus "restarts regularly" points away from the application. Get the exit code, which is the actual signal.
kubectl describe pod <pod> | grep -A6 "Last State"
Exit code
Meaning
Where the fault is
137
SIGKILL — almost always OOM
Memory limit, or an unhandled SIGTERM timing out
143
SIGTERM — something asked it to stop
The liveness probe, or an eviction
0
It completed successfully
Wrong command; this should be a Job
139
Segfault
Wrong CPU architecture (amd64 image on arm64)
137 and 143 both mean the platform killed it, not that the app failed — which is exactly why the logs are clean. The app never got to log an error.
# 137: was it OOM? Check the limit against real usage
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}{"\n"}'
kubectl top pod <pod>
# 143: is a probe killing it?
kubectl describe pod <pod> | grep -i -A3 probe
One service cannot reach another. dig works from inside the pod. · dig شغّال والتطبيق مش بيوصل
dig succeeding while the application fails is a diagnosis, not a paradox — they use different code paths.
dig → asks the resolver DIRECTLY
skips /etc/hosts, skips search-domain expansion
your app → getaddrinfo()
/etc/hosts → /etc/nsswitch.conf → search domains → DNS
# Compare the two. The difference IS the answer.
dig myservice.default.svc.cluster.local +short
getent hosts myservice # same path the app uses
cat /etc/resolv.conf # search domains and ndots
cat /etc/hosts # a stale override wins over DNS
dig
getent
Where the fault is
✔
✔
Not name resolution. Look at the port, the Service, or a NetworkPolicy
✔
✘
The system lookup chain — ndots, search domains, nsswitch
✘
✘
Genuine DNS failure. Check CoreDNS pods and egress to port 53
✘
✔
An /etc/hosts entry is shadowing the name
Then, if resolution is fine, the layers below it:
nc -zv myservice 8080 # refused = nothing listening; timeout = blocked
kubectl get endpoints myservice # <none> = selector mismatch or not Ready
kubectl get networkpolicy -A # policies apply to the DESTINATION
A file upload endpoint hangs at the same point every time. Small requests are fine. · الملفات الكبيرة بتتعلّق والصغيرة تمام
Same-point hangs with size dependence is MTU / path MTU discovery, not bandwidth and not the application.
Small packets fit any link. Once a packet exceeds the smallest MTU on the path, a router must fragment it or reply ICMP fragmentation needed. If a firewall blocks that ICMP, the sender never learns and retransmits the oversized packet forever — a PMTUD black hole.
client (MTU 1500) ──→ tunnel (MTU 1400) ──→ server
│ │
│ handshake, small │ fits — everything looks healthy
│ 1500-byte payload │ too big
│ │
│ ←── ICMP "frag needed, use 1400"
│ ✘ dropped by a firewall
│
└── never learns. retransmits forever. hangs at the same offset.
# Find the real path MTU. -M do = do not fragment
ping -M do -s 1472 <host> # 1472 + 28 bytes of header = 1500
ping -M do -s 1372 <host> # step down until it succeeds
ip link show # what MTU is actually configured
Where it actually happens: VPNs and tunnels (WireGuard, IPsec), and overlay networks — including several Kubernetes CNIs, which encapsulate packets and so reduce the usable MTU below 1500. The signature never varies: handshakes and small responses fine, large payloads hang.
A server is at 100% CPU. The application team says their code did not change. · ١٠٠٪ معالج والكود ما اتغيرش
Do not open top and stop. top tells you who; it does not tell you why, and the four vmstat columns are where the answer is.
# 1. Is it genuinely CPU-bound? Load must be read against core count.
uptime; nproc
# load 8 on 4 cores = saturated
# load 0.5 with "100%" = you are reading a single-core spike
# 2. WHERE is the time going? This is the decisive command.
vmstat 1 5
Column
High means
Which is a completely different problem
us
User time
Application code. The usual assumption
sy
System time
Kernel: syscalls, network, sometimes a container escaping its cgroup
wa
I/O wait
Not CPU at all. You are blocked on disk
st
Steal
The hypervisor is throttling you. Not your fault, not fixable here
# 3. Only now, find the process
ps aux --sort=-%cpu | head
# 4. And what it is doing
strace -c -p <pid> # syscall counts. Cheap and often decisive
Disk is full. df says 95%, du -sh / says 40%. · df بيقول مليان و du بيقول فاضي
Neither is lying. A deleted file is still held open by a process.
du walks the directory tree and sums files it can see. df asks the filesystem how many blocks are allocated. When a process holds a file open and someone rms it, the directory entry is gone — so du cannot see it — but the blocks are not freed until the last descriptor closes.
lsof +L1 # files with link count 0, still open
lsof -nP | grep '(deleted)'
# Reclaim without a restart — truncate through the descriptor
: > /proc/<pid>/fd/<fd>
The usual cause is log rotation that rms the log instead of truncating it, while the application keeps writing to the same open descriptor. The fix is copytruncate, or a SIGHUP to make the service reopen its files.
Everything broke at 02:00 and nobody deployed. Where do you start? · كل حاجة باظت الساعة ٢ ومحدش نشر حاجة
"Nobody deployed" removes the most likely cause, so go to the things that change without a deploy. A precise timestamp is a strong clue: it points at something scheduled or something expiring.
# The single most valuable command when many things break at once —
# it orders everything by time and usually reveals ONE cause
kubectl get events -A --sort-by=.lastTimestamp | tail -40
journalctl --since "01:50" --until "02:20" -p warning
Then work the list of things that change on their own:
Suspect
How to check
Why 02:00
Certificate expiry
`echo \
openssl s_client -connect host:443 2>/dev/null \
openssl x509 -noout -dates`
Certs expire at a precise time
Cron / scheduled job
crontab -l, kubectl get cronjobs -A
Backups and batch jobs run at night
Log rotation
/etc/logrotate.d/, plus the df/du case above
Runs nightly, can fill or free disk
Token / secret rotation
Cloud IAM logs, secret manager audit
Often scheduled
Disk filling gradually
df -h, df -i
Crosses the threshold at a random-looking time
A dependency's deploy
Their status page, upstream changelog
You did not deploy — they did
DNS TTL expiry
dig the record; compare to authoritative
A change hours earlier lands when caches expire
A database is slow. The team wants to add read replicas. What do you say? · الداتابيز بطيئة والفريق عايز replicas
"Let's confirm the bottleneck is read capacity first." Adding replicas is a real fix for a real problem — but only for read throughput, and it makes several other problems worse.
read replicas HELP read replicas DO NOT HELP
────────────────── ─────────────────────────
too many concurrent reads a missing index
read-heavy analytics on the write contention / lock waits
primary a single slow query
geographic read latency connection-pool exhaustion
N+1 queries from the app
│
└─ and replicas ADD
replication lag, so
reads can now be stale
Investigate in this order, because it is cheapest-first and each step can end the conversation:
Find the slow queries.pg_stat_statements ordered by total time. One query is usually most of the load.
Check for a missing index.EXPLAIN ANALYZE on the worst one. A sequential scan on a large table is a five-minute fix, not a new server.
Reads or writes? If waits are on locks or write I/O, replicas change nothing.
Connection pool. Thousands of idle connections each cost memory; a pooler often fixes "slow" outright.
Then, if it is genuinely read throughput, add replicas — and say out loud that the application must now tolerate replication lag, so read-after-write flows need to go to the primary.