DNS turns a name into an address. Every answer is cached with an expiry, and there is no way to recall a cached answer early — which explains almost every DNS problem you will meet.
DNS (Domain Name System) is the internet's phone book. You type api.example.com; DNS returns 93.184.216.34; your browser connects to that address.
It exists because addresses change and names should not. Move your service to a new server and you update one DNS record instead of reconfiguring every client.
No single server knows every name. Each level only knows who to ask next.
your app
│ getaddrinfo("api.example.com")
↓
STUB RESOLVER reads /etc/hosts, then /etc/resolv.conf
│
↓
RECURSIVE RESOLVER your ISP, 8.8.8.8, or CoreDNS in a cluster
│ does the walking, caches every answer
├──→ ROOT servers "who handles .com?"
├──→ .com servers "who handles example.com?"
└──→ example.com servers "here is the A record"
│
↓
answer + TTL ←── cached at every layer on the way back
The resolver does the work; your machine just asks it once.
Every record carries a TTL (time to live) in seconds: how long resolvers may cache it.
you change the record
│
│ resolvers holding the OLD answer keep serving it
│ for up to TTL seconds. There is no recall.
↓
t=0 ──────────── TTL ────────────→ everyone converged (in theory)
# 1. dig — asks the RESOLVER directly. Best for inspecting DNS itself.
dig +short api.example.com
# 2. getent hosts — goes through the SAME path your application uses,
# including /etc/hosts and search domains.
getent hosts api.example.com
# 3. dig +trace — walk the delegation from the root. Use when you suspect
# the zone itself is misconfigured.
dig +trace api.example.com
dig api.example.com
# ;; ANSWER SECTION:
# api.example.com. 300 IN A 93.184.216.34
# └┬┘ └┬┘ └─────┬────┘
# │ │ └── the address
# │ └── record type
# └── TTL remaining, in seconds
Watching the TTL count down on repeated queries tells you the answer is cached. A TTL that resets to its full value means you reached the authoritative server.
DNS uses UDP port 53 for normal queries because it is fast and queries are tiny. But if a response exceeds the advertised size limit, the server sets a truncation flag and the resolver retries over TCP port 53.
query ──→ UDP/53 ──→ response fits? ── yes ──→ done
│
no (large answer)
↓
retry over TCP/53
A firewall that allows UDP/53 but blocks TCP/53 produces a system that works perfectly until an answer grows — typically when a service scales up and its address list gets longer. It then fails for exactly the records that matter most, and the failure looks random.
You changed a DNS record 30 minutes ago. Some users get the new server, some the old. Explain and fix.
This is normal, expected DNS behaviour, and saying so is the first half of the answer.
Why: DNS is a cache hierarchy with no invalidation. Every resolver that queried before your change holds the old answer for the remainder of its TTL, and there is no protocol mechanism to recall it.
# What the authoritative server says now — the source of truth
dig @ns1.example.com api.example.com
# What a public resolver still has, and its remaining TTL
dig @8.8.8.8 api.example.com
What you can do: wait out the TTL. That is genuinely it, for DNS.
What you should have done: lowered the TTL a full old-TTL period before the change.
In a Kubernetes cluster, DNS lookups are slow and CoreDNS CPU is high. Where do you look?
Almost always ndots and search-domain expansion.
A pod's /etc/resolv.conf contains options ndots:5 and several search domains. ndots:5 means: any name with fewer than 5 dots is tried against every search domain first, before being tried as-is.
your app connects to "api.example.com" (2 dots, under ndots:5)
the resolver tries, in order:
api.example.com.default.svc.cluster.local ✘ NXDOMAIN
api.example.com.svc.cluster.local ✘ NXDOMAIN
api.example.com.cluster.local ✘ NXDOMAIN
api.example.com ✔ finally
4 queries instead of 1 — and each ✘ is also an IPv6 (AAAA)
query, so realistically 8 round trips for one connection
Multiply by every connection from every pod and CoreDNS is doing an order of magnitude more work than it should.
Fixes, in order of preference:
Fix
How
Fully qualify external names
Use api.example.com. with a trailing dot — no expansion
NodeLocal DNSCache — a per-node resolver, removes most cluster-wide traffic
Why can a CNAME not exist at the domain apex, and what do you use instead?
Because the spec forbids a CNAME from coexisting with any other record for the same name — and a zone apex is required to have SOA and NS records. So example.com can never be a CNAME, while www.example.com can.
What to use instead: a provider-specific pseudo-record that behaves like a CNAME but returns an address:
Provider
Mechanism
Route 53
ALIAS record
Cloudflare
CNAME flattening
Azure DNS
ALIAS record
Generic fallback
An A record with a static IP — only if the target IP is genuinely stable