Gendi Notes

Kubernetes Services

Pods get a new IP address every time they restart, so nothing can talk to them directly. A Service is the stable address that stays put.

Level 2 guide kubernetesservicesnetworking

What is it?#

A Service gives a group of pods one fixed IP address and DNS name. Clients talk to the Service; it forwards to whichever pods are currently healthy.

without a Service with a Service ───────────────── ────────────── client → 10.1.0.4 ✗ gone client → web (10.96.0.10) → 10.1.0.9 ✗ gone │ → 10.1.2.3 ✓ for now ├→ 10.1.0.4 ├→ 10.1.0.5 who tracks the current list? └→ 10.1.2.6 every client, forever. Kubernetes tracks it. Once.

Why it exists#

Pod IPs are ephemeral by design. A pod that restarts, moves node, or scales gets a different address. Hard-coding one guarantees breakage.

A Service solves two problems at once: a stable address, and load balancing across whatever pods exist right now.

What it is made of#

Two things happen behind one object — and nearly all Service bugs are in the first while people debug the second:

① the ENDPOINT list ② the ROUTING rules ───────────────────── ─────────────────── "which pods match my "rewrite traffic for selector AND pass 10.96.0.10 to one of their readiness probe?" those pod IPs" maintained by the programmed into each node's endpoints controller kernel by kube-proxy

The four types#

TypeGives youReachable fromUse when
ClusterIP (default)Internal IP + DNS nameInside the cluster onlyService-to-service. Most of the time
NodePortA port on every nodeOutside, if you can reach nodesRarely direct — it is a building block
LoadBalancerA cloud load balancerThe internetCloud-managed external entry
ExternalNameA DNS CNAME, no proxyingInsideAliasing an external hostname

How to use it#

service.yaml
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: ClusterIP
  selector:
    app: web          # MUST match the pods' labels exactly
  ports:
    - port: 80        # the port the Service listens on
      targetPort: 8080  # the port the CONTAINER listens on

DNS names#

Once a Service exists, it has a name:

web . production . svc . cluster.local │ │ │ │ │ │ │ └─ cluster domain │ │ └─ "svc" for Services │ └─ namespace └─ Service name
FromYou can write
Same namespaceweb
Another namespaceweb.production
Anywhere (fully qualified)web.production.svc.cluster.local

So an app connects to postgres://db:5432 where db is just the Service name. You never handle IP addresses.

Commands#

kubectl get svc                      # all Services with their ClusterIPs
kubectl get endpoints web            # THE FIRST DEBUGGING COMMAND
kubectl describe svc web

# Test from inside the cluster — a Service is not reachable from your laptop
kubectl run tmp --rm -it --image=busybox --restart=Never -- sh
#   inside: wget -qO- http://web
#   inside: nslookup web

# Or reach it from your machine temporarily
kubectl port-forward svc/web 8080:80    # then open http://localhost:8080

kubectl port-forward is how you test an internal Service from your laptop without exposing it to the internet.

What goes wrong#

SymptomCauseFix
ENDPOINTS: <none>Selector mismatch, or pods not ReadySee above
Connection refusedtargetPort wrongCompare to the container's real port
ping <ClusterIP> failsA ClusterIP is not pingable — it is a routing rule, not a hostUse wget/curl, not ping
Works from one pod, not anotherNetworkPolicy blocking itkubectl get networkpolicy -A
LoadBalancer stuck on <pending>No cloud integration (e.g. plain minikube)Use port-forward or NodePort locally
Scaled up, load still unevenKeep-alive connections pinned to old podsSee below

Getting traffic in from outside#

A ClusterIP Service is internal. For HTTP from the internet you normally want an Ingress in front of it — one load balancer for many services, with hostname and path routing and TLS.

internet │ ↓ Ingress example.com/api → api Service │ example.com/ → web Service ├──→ Service (api) ──→ pods └──→ Service (web) ──→ pods
ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: site
spec:
  ingressClassName: nginx      # which controller handles this
  rules:
    - host: example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port: { number: 80 }

An Ingress object does nothing on its own — a controller (ingress-nginx, Traefik) must be installed in the cluster to act on it. An Ingress that appears to be ignored usually means no controller, or the wrong ingressClassName.

Interview corner · الأسئلة اللي بتتسأل#

How does a Service actually route traffic? Is it a proxy? · الـ Service بروكسي؟

No, and this is the answer that shows you understand the data path.

A ClusterIP is a virtual IP that no process listens on. There is no Service pod, no Service container, nothing to SSH into. What exists is a set of packet-rewriting rules in every node's kernel.

Pod sends a packet to 10.96.0.10:80 (the ClusterIP) │ ↓ node kernel: iptables / IPVS / eBPF rules, written by kube-proxy │ DNAT — rewrite the destination to a real pod IP ↓ 10.1.0.5:8080 ← one of the Ready endpoints, picked here │ ↓ conntrack records the choice, so every later packet of the same connection goes to the SAME pod

Three things this explains immediately:

  • You cannot ping a ClusterIP. There is no host there to reply. A failing ping proves nothing at all.
  • Load balancing is per connection, because the pod is chosen once at DNAT time and conntrack pins it.
  • kube-proxy is not in the request path. It only writes the rules. Kill kube-proxy and existing traffic keeps flowing; only new Services stop being programmed.
A rolling deploy causes a burst of 502s every time, even though the new pods are healthy. Why?

Almost always the de-registration race: the pod stops accepting traffic after it has already been sent some.

When a pod is deleted, two things happen in parallel, not in order:

pod deleted │ ├──→ kubelet sends SIGTERM to the container ← starts dying NOW │ └──→ endpoints controller removes it from the EndpointSlice, then EVERY node must reprogram its rules ← takes 100ms-2s In that gap the pod is shutting down but nodes still route to it. Those requests become 502s.

The fix is a preStop hook that just sleeps:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]

That looks absurd and is correct. It delays SIGTERM long enough for the endpoint removal to propagate everywhere, so the pod keeps serving during the window when nodes might still route to it. Combine with an application that handles SIGTERM by draining rather than exiting immediately.

When would you use a headless Service, and what does it change?

Set clusterIP: None and you get a Service with no virtual IP and no routing rules. DNS then returns the pod IPs directly — one A record per Ready pod — instead of a single ClusterIP.

Normal ClusterIPHeadless (clusterIP: None)
DNS returnsone virtual IPevery Ready pod's IP
Load balancingkernel, per connectionthe client decides
Individual pods addressablenoyes, via StatefulSet DNS names

Use it when the client needs to know about individual pods:

  • StatefulSets — a database replica set where you must reach db-0 specifically, because it is the primary.
  • gRPC client-side load balancing — the client gets all the addresses and balances per RPC, which fixes the keep-alive pinning problem properly.
  • Peer discovery — clustered software (Cassandra, Kafka, Elasticsearch) where members find each other.

Key takeaways#

  • Pod IPs change; a Service is the stable address.
  • kubectl get endpoints <svc> first, always. <none> means selector mismatch or pods not Ready.
  • port is the Service's, targetPort is the container's.
  • A ClusterIP is not pingable — that failing proves nothing.
  • Services are reachable from inside the cluster; use port-forward from your laptop.
  • Load balancing is per connection. Keep-alive defeats it.
  • Ingress for HTTP from outside — and it needs a controller installed.

Last updated 2026-09-08