DNS

In one line

DNS (Domain Name System) is a distributed, cached, hierarchical database that maps names to records. A recursive resolver walks the tree from the root down on your behalf and caches every answer for its TTL (time to live).

Why it exists

Early ARPANET hosts kept one file, HOSTS.TXT, maintained centrally and downloaded by every machine. Every new host meant a new copy for everyone, names had to be globally unique in a flat space, and one team was a bottleneck for the whole network.

Paul Mockapetris designed DNS in 1983 (RFC 882 and 883) and the current base specification followed in November 1987 (RFC 1034 and 1035). Three ideas fixed the file:

Hierarchy. The name space is a tree. www.example.com. is read right to left: root (.), com, example, www.
Delegation. Each node can hand a subtree (a zone) to someone else with NS records. Nobody owns the whole database.
Caching. Every record carries a TTL, so answers are reused and the top of the tree is rarely asked.

Later layers patched what 1987 did not foresee: larger messages (EDNS0, 1999, current RFC 6891 from 2013), authenticity (DNSSEC, 2005), privacy (DNS over TLS in 2016, DNS over HTTPS in 2018, DNS over QUIC in 2022), and service metadata (SVCB and HTTPS records, 2023).

How it works

The resolution path

Example: an application asks for the A record of example.com and nothing is cached. This is step 4 of What happens when you open a website.

Stub resolver. The application calls getaddrinfo(). The libc stub checks /etc/hosts, then sends the query to a nameserver from /etc/resolv.conf with the RD (recursion desired) flag set. The stub does no tree walking. glibc defaults: 5 second timeout, 2 attempts, at most 3 nameservers, ndots:1.
Recursive resolver. Your ISP, your VPC, 1.1.1.1, 8.8.8.8, or CoreDNS plus its upstream. It answers from cache if it can. Otherwise it walks the tree itself.
Root. The resolver asks a root server. The root does not know the answer. It returns a referral: the NS records for com plus glue (the A and AAAA records of those nameservers). There are 13 root server names, a to m.root-servers.net, run by 12 operators, each name anycast from many sites. Resolvers ship with their addresses in a hints file.
TLD. The resolver asks a com server. It returns a referral to the nameservers of example.com.
Authoritative. The resolver asks a nameserver of example.com. It answers with the record and the AA (authoritative answer) flag.
Cache and reply. The resolver caches each referral and the final answer for their TTLs and replies to the stub. The next lookup for anything under com skips the root. The next lookup for the same name skips everything.

The stub asks one question and waits. The recursive resolver does the walking: two referrals, then the authoritative answer. Only the green arrows carry the final record.

With QNAME minimisation (RFC 9156) the resolver sends each level only the labels it needs, for example com NS to the root, so upstream servers do not learn the full name.

The message

One format for queries and responses: a 12 byte header, then Question, Answer, Authority and Additional sections.
Header flags worth knowing: QR (response), AA (authoritative answer), TC (truncated), RD (recursion desired), RA (recursion available), AD (authentic data, DNSSEC validated), CD (checking disabled).
A label is at most 63 bytes. A full name is at most 255 bytes.
A lookup key is the triple name, type, class. Class is IN (Internet) in practice.

UDP 53, TCP 53 and truncation

A query goes over UDP port 53 by default. One datagram out, one back, no handshake.
The original limit for a UDP DNS message is 512 bytes.
If the answer does not fit, the server sends what fits and sets the TC flag.
The client retries the same query over TCP port 53. Each message is prefixed with a 2 byte length. RFC 7766 (2016) makes TCP support mandatory for every general purpose implementation, so “DNS is UDP only” firewall rules are wrong.
TCP is also used for zone transfers (AXFR and IXFR).

EDNS0

EDNS0 (Extension Mechanisms for DNS, RFC 6891) adds a pseudo record of type OPT to the Additional section. It carries:

the UDP payload size the sender can accept, so answers larger than 512 bytes fit in UDP. A value below 512 is treated as 512;
the DO bit (DNSSEC OK), which asks for signatures;
an extended RCODE, a version (0) and options such as Client Subnet and cookies.

RFC 6891 suggests 4096 bytes as a starting value. Large UDP answers get fragmented at the IP layer, and fragments are dropped by many firewalls. DNS Flag Day 2020 (1 October 2020) recommended 1232 bytes: the IPv6 minimum MTU of 1280 minus 48 bytes of IPv6 and UDP headers. Anything bigger is truncated and retried over TCP.

TTL and negative caching

Every record has a TTL in seconds, set by the zone owner. A resolver may cache the record for at most that long and counts it down. dig shows the remaining time when it asks a cache and the full value when it asks the authoritative server.
A change is visible to everyone only after the old TTL runs out. Lower the TTL first, wait out the old TTL, then make the change.
Negative answers are cached too (RFC 2308). There are two kinds:
- NXDOMAIN: the name does not exist at all.
- NODATA: the name exists but has no record of that type. It is NOERROR with an empty answer section.
The negative TTL is the minimum of the SOA record’s TTL and the SOA MINIMUM field. The SOA record is returned in the Authority section so the resolver can read both.
A resolver may cache a server failure (SERVFAIL) for at most 5 minutes.

Practical effect: if you query a name before you create it, the “does not exist” answer can stick for the negative TTL.

DNSSEC in brief

DNSSEC (DNS Security Extensions, RFC 4033 to 4035) adds origin authentication: a validating resolver can prove an answer came from the zone owner and was not changed. It does not encrypt anything.

The zone signs each record set. The signature is an RRSIG record.
The public keys are in DNSKEY records. Usually a zone signing key (ZSK) signs the data and a key signing key (KSK) signs the DNSKEY set.
The parent zone publishes a DS record: a hash of the child’s KSK. This links child to parent.
The chain continues up to the root, whose key is the trust anchor configured in the resolver.
NSEC or NSEC3 records prove that a name or type does not exist.
A client sets the DO bit to get signatures. A validating resolver sets the AD flag on answers it verified. If validation fails it returns SERVFAIL.

Costs: bigger answers (hence EDNS0 and TCP), key rollovers, and a hard failure mode. An expired signature or a stale DS record takes the whole zone offline for validating resolvers.

Encrypted transports

Plain DNS is readable and forgeable on the path. Three transports encrypt the hop from stub to recursive resolver. See the table in Compare.

Anycast

The same IP prefix is announced with BGP from many locations. Routers send each packet to the nearest announcement. One address, many servers.

The root servers, the large TLDs and public resolvers such as 1.1.1.1 and 8.8.8.8 all use it.
It gives low latency, spreads DDoS load across sites, and removes a site by withdrawing its route.
It suits DNS well: one UDP question, one answer, no long-lived state to break when routing shifts.

Split-horizon

Split-horizon (split-brain) DNS gives different answers for the same name depending on who asks. Typical uses: db.corp.example.com resolves to a private address inside the VPC or VPN and does not exist outside, or app.example.com resolves to an internal load balancer inside and a CDN outside.

Mechanisms: BIND views, a private hosted zone attached to a VPC, a conditional forwarder for one domain, or a cluster resolver that answers a private zone itself.

The cost is debugging. “Works for me” now depends on which resolver each side used.

Kubernetes DNS

Cluster DNS is CoreDNS, behind a Service that is still named kube-dns in kube-system. kubelet writes /etc/resolv.conf into every pod:

nameserver 10.32.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Records CoreDNS serves:

Object	Name	Answer
Service (ClusterIP)	`my-svc.my-namespace.svc.cluster.local`	A or AAAA: the cluster IP
Headless Service	same name	A or AAAA: the IPs of all ready pods
Named port	`_port-name._port-protocol.my-svc.my-namespace.svc.cluster.local`	SRV: port number and target name
Pod with `hostname` and `subdomain`	`hostname.subdomain.my-namespace.svc.cluster.local`	A or AAAA: the pod IP

Why lookups multiply. ndots:5 means: if a name has fewer than 5 dots, try it with every search domain first, and only then as written. The default outside Kubernetes is ndots:1.

A pod in namespace shop resolves api.example.com (2 dots):

api.example.com.shop.svc.cluster.local.   NXDOMAIN
api.example.com.svc.cluster.local.        NXDOMAIN
api.example.com.cluster.local.            NXDOMAIN
api.example.com.                          NOERROR

That is 4 names. glibc asks for A and AAAA, so one external lookup costs 8 queries, 6 of them wasted. Cloud providers often append their own search domains, which adds more. In exchange, short names work: my-svc and my-svc.other-namespace resolve from any pod.

Ways to cut the waste:

Write external names with a trailing dot: api.example.com. is absolute and skips the search list.
Lower ndots for a workload with dnsConfig.options in the pod spec.
Run NodeLocal DNSCache (stable since Kubernetes v1.18): a caching agent on every node as a DaemonSet, listening on a link-local address. Pods skip the DNAT and conntrack entry for the Service IP, and the agent talks to CoreDNS over TCP.
The CoreDNS autopath plugin completes the search path on the server. It needs pods verified.

Other defaults that matter:

dnsPolicy defaults to ClusterFirst. A pod with hostNetwork: true falls back to the node’s resolver unless you set ClusterFirstWithHostNet. The policy named Default means “inherit from the node” and is not the default.
The kubeadm Corefile uses kubernetes cluster.local in-addr.arpa ip6.arpa with ttl 30, forward . /etc/resolv.conf for everything else, and cache 30. Without the ttl line the kubernetes plugin serves a TTL of 5 seconds.
dnsConfig accepts at most 3 nameservers. Kubernetes allows up to 32 search domains and 2048 characters in total.

Compare

Who is who

Role	What it does	Holds data	Examples
Stub resolver	sends one recursive query, waits	small cache or none	glibc, musl, systemd-resolved stub, browser
Forwarder	passes queries to another resolver, caches	cache only	dnsmasq, CoreDNS `forward`, home router
Recursive resolver	walks the tree, caches, may validate DNSSEC	cache only	Unbound, BIND, Knot Resolver, 1.1.1.1, 8.8.8.8
Authoritative server	answers for zones it holds, never recurses	the zone itself	BIND, NSD, Knot DNS, PowerDNS, Route 53, Cloud DNS
Root and TLD servers	authoritative servers that mostly return referrals	delegations	`a` to `m.root-servers.net`, the `com` servers

Record types

Type	Holds	Example	Remember
A	IPv4 address	`example.com. 300 IN A 192.0.2.10`	several A records give round-robin
AAAA	IPv6 address	`example.com. 300 IN AAAA 2001:db8::10`	asked in parallel with A
CNAME	alias to another name	`www 300 IN CNAME example.com.`	cannot share a name with other records, so not allowed at the zone apex
NS	nameserver of a zone	`example.com. IN NS ns1.example.com.`	exists in the parent (delegation) and in the child; they must agree
SOA	start of authority	primary NS, contact, serial, refresh, retry, expire, minimum	one per zone; drives zone transfers and the negative TTL
MX	mail exchanger	`example.com. IN MX 10 mail.example.com.`	lower preference wins; the target must be a name with A or AAAA, not a CNAME
TXT	free text	`"v=spf1 include:_spf.example.com ~all"`	SPF, DKIM, DMARC, domain ownership checks; each string is at most 255 bytes
SRV	service location	`_sip._tcp 300 IN SRV 10 60 5060 sip.example.com.`	priority, weight, port, target; used by Kubernetes named ports
PTR	reverse lookup	`10.2.0.192.in-addr.arpa. IN PTR host.example.com.`	IPv6 uses `ip6.arpa`; mail servers check it
CAA	which CAs may issue certificates	`example.com. IN CAA 0 issue "letsencrypt.org"`	tags `issue`, `issuewild`, `iodef`; the CA climbs the tree to the first match; no record means no restriction
SVCB, HTTPS	service binding (types 64 and 65)	`example.com. IN HTTPS 1 . alpn="h3,h2"`	priority 0 is an alias that works at the apex; other values carry `alpn`, `port`, `ipv4hint`, `ipv6hint` and the ECH key

The HTTPS record lets a browser learn about HTTP/3 and Encrypted Client Hello (TLS) before it opens any connection.

Transports

	Plain DNS (Do53)	DoT	DoH	DoQ
Full name	DNS over UDP and TCP	DNS over TLS	DNS over HTTPS	DNS over QUIC
Specification	RFC 1035 (1987), RFC 7766	RFC 7858 (2016)	RFC 8484 (2018)	RFC 9250 (2022)
Port	UDP 53, TCP 53	TCP 853	TCP 443 (HTTP/2 or later recommended)	UDP 853
Encrypted	no	yes	yes	yes
Framing	raw message, 2 byte length on TCP	2 byte length inside TLS	`application/dns-message` in a GET or POST	one QUIC stream per query
Head-of-line blocking	none on UDP	yes, one TCP stream	yes on HTTP/2, no on HTTP/3	no
Easy for a network to block	yes	yes, dedicated port	hard, looks like web traffic	yes, dedicated port
Setup cost	none	TCP plus TLS handshake, then reuse	TCP plus TLS plus HTTP, then reuse	one QUIC handshake, then reuse

All four protect only the hop to the resolver. The resolver still sees every query. DNSSEC is a separate axis: it proves the data is authentic and works over any of these transports.

Response codes

RCODE	Meaning	Usual cause
`NOERROR` with answers	success
`NOERROR`, empty answer (NODATA)	name exists, no record of that type	asking AAAA for an IPv4-only name
`NXDOMAIN`	the name does not exist	typo, record not created, wrong search domain
`SERVFAIL`	the resolver could not get a valid answer	DNSSEC validation failed, all authoritative servers unreachable, lame delegation
`REFUSED`	the server will not answer you	recursion not allowed for your address, zone not served there
no reply, timeout	the packet was lost or dropped	firewall, wrong resolver IP, UDP conntrack problem

What breaks

Symptom	Cause	Check
Name resolves for some users, not others	old record still cached until the TTL ends	`dig example.com` twice, watch the TTL count down; `dig @ns1.example.com example.com` for the truth
New record returns `NXDOMAIN` for minutes	negative caching of an earlier miss	`dig example.com SOA`, read the SOA TTL and the last field (minimum)
`SERVFAIL` from a validating resolver, works with `+cd`	broken DNSSEC: expired RRSIG or stale DS at the parent	`dig +dnssec example.com`, `dig +cd example.com`, `dig DS example.com`
`SERVFAIL` or slow, intermittent	lame delegation: the parent lists a nameserver that does not serve the zone	`dig +trace example.com`, `dig NS example.com @a.gtld-servers.net`
Large answers fail, small ones work	TCP 53 blocked, or fragments dropped	`dig +tcp example.com TXT`, `dig +notcp +bufsize=1232 example.com DNSKEY`, look for `tc` in flags
Different answer inside and outside	split-horizon, or a VPN pushed another resolver	`dig @10.0.0.2 name`, `dig @1.1.1.1 name`, `resolvectl status`
Certificate issuance refused	CAA record names another CA	`dig example.com CAA`
Zone file rejects `CNAME` at `example.com`	CNAME cannot coexist with SOA and NS at the apex	use the provider’s alias record or an HTTPS record in alias mode
Pod: external names slow	`ndots:5` search expansion	`kubectl exec POD -- cat /etc/resolv.conf`, enable the CoreDNS `log` plugin, add a trailing dot
Pod: random 5 second delays	a UDP query was lost (for example a conntrack race on the Service DNAT) and glibc waited its 5 second timeout	NodeLocal DNSCache, `conntrack -S` on the node (`insert_failed`)
Pod: nothing resolves	CoreDNS down, no endpoints, or a NetworkPolicy blocks UDP and TCP 53	commands below
Pod on Alpine 3.17 or earlier fails on big answers	old musl had no TCP fallback, so answers over 512 bytes failed	use Alpine 3.18 or later
CoreDNS pods crash-loop with “Loop detected”	the node’s `/etc/resolv.conf` points at a local stub (systemd-resolved), so CoreDNS forwards to itself	point kubelet `--resolv-conf` at `/run/systemd/resolve/resolv.conf`

Read a dig answer:

dig example.com A

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4242
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; ANSWER SECTION:
example.com.        287     IN      A       192.0.2.10
;; Query time: 12 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)

status is the RCODE. aa in flags means the server is authoritative. ad means DNSSEC validated. tc means truncated. 287 is the TTL left in the cache.

The commands worth memorising:

dig +short example.com                 # the answer only
dig @1.1.1.1 example.com AAAA          # ask a specific resolver
dig +trace example.com                 # walk root, TLD, authoritative yourself
dig +norecurse @ns1.example.com example.com   # what the authoritative server says
dig -x 192.0.2.10                      # reverse lookup (PTR)
dig +tcp example.com                   # force TCP 53
dig +dnssec example.com                # show RRSIG records, check the ad flag
dig +cd example.com                    # skip validation: works here but fails without it means DNSSEC
dig example.com HTTPS                  # HTTPS record: alpn, hints, ECH
dig example.com SOA +multiline         # serial, refresh, retry, expire, minimum

Kubernetes, from the official debugging guide:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
kubectl exec -i -t dnsutils -- cat /etc/resolv.conf
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl get svc -n kube-system kube-dns
kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns
kubectl -n kube-system edit configmap coredns     # add "log" to the .:53 block to see every query

Keywords

DNS, stub resolver, recursive resolver, forwarder, authoritative server, root servers, root hints, TLD, zone, delegation, referral, glue record, FQDN, trailing dot, A, AAAA, CNAME, apex, NS, SOA, serial, MX, TXT, SPF, DKIM, DMARC, SRV, PTR, in-addr.arpa, CAA, SVCB, HTTPS record, TTL, negative caching, NXDOMAIN, NODATA, SERVFAIL, REFUSED, UDP 53, TCP 53, TC bit, truncation, EDNS0, OPT record, 1232 bytes, DNS Flag Day, QNAME minimisation, DNSSEC, RRSIG, DNSKEY, DS, NSEC, NSEC3, KSK, ZSK, DO bit, AD flag, DoT, DoH, DoQ, port 853, anycast, split-horizon, CoreDNS, kube-dns, ndots:5, search domains, resolv.conf, dnsPolicy, ClusterFirst, dnsConfig, NodeLocal DNSCache, autopath, conntrack, dig +trace.