I’ve been handling DNS resolution problems in Kubernetes recently — a good chance to learn how the DNS server in Kubernetes works. The dns server problem I handled will get its own blog later.
My understanding of the resolution process is fairly shallow; I’ll only introduce the configuration contents.
DNS Overview in a Pod
As everyone knows, DNS servers convert domain names to IPs (for why the conversion is needed, I suggest reviewing the 7-layer network model).
On Linux servers, dns resolution configuration lives in /etc/resolv.conf — pods are no exception. Here’s the configuration in a certain Pod:
1 | nameserver 10.96.0.10 |
Suppose we normally want to modify our own machine’s dns server, e.g. to 8.8.8.8, we’d modify it like this:
1 | nameserver 8.8.8.8 |
To debug a DNS server and test returned results, you can use the dig tool:
1 | > dig baidu.com @8.8.8.8 |
The DNS Server – nameserver
Let’s start with nameserver 10.96.0.10 — why does requesting this address perform DNS resolution?
The answer is iptables. I excerpt only udp port 53; the following can be obtained via iptables-save.
1 | -A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU |
Let’s look at this chain KUBE-SVC-TCOU7JCQXEZGVUNU:
1 | -A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-Q3HNNZPXUAYYDXW2 |
Kubernetes’ deployment
Now look at our kubernetes pods’ IP addresses — that is, dns requests actually reach our coredns containers to be processed.
1 | > kubectl -n kube-system get pods -o wide | grep dns |
The Concrete Implementation of Services in Kubernetes
Checking the corresponding service, you can see the iptables on the machines above are actually the concrete implementation of the service.
1 | > kubectl -n kube-system get svc | grep dns |
Some may wonder: now 2 pods split traffic evenly — what about 3 or 4 pods? How does iptables forward then?
I happened to have this question, so I added 2 more pods to see how iptables splits traffic evenly across 4 pods.
This is the final implementation:
1 | -A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-HTZHQHQPOHVVNWZS |
These statements should mean:
- The first 1/4 of traffic goes to one chain, leaving 3/4
- Of the remaining 3/4, 1/3 goes to one chain, leaving 2/4
- Of the remaining 2/4, 1/2 goes to one chain, leaving 1/4
- The last 1/4 goes to one chain
Traffic is evenly split this way — quite clever. This way, 5 or 10 pods can also be divided successively.
Parsing Other resolv.conf Parameters
1 | search kube-system.svc.cluster.local svc.cluster.local cluster.local |
For a detailed introduction see here: resolv.conf manual. Let me briefly state my understanding.
The search Parameter
Without this search parameter, when we look up:
1 | > ping kube-dns |
After adding the search parameter, look up again:
1 | > ping kube-dns |
You can see that when resolving a domain, if the given domain can’t be found, the suffixes after search are appended for lookup (if it ends with ., like kube-dns., such FQDN domains won’t be retried).
search‘s job is trying for us. Used in kubernetes, configuring kube-system.svc.cluster.local svc.cluster.local cluster.local
means when we ping abc, queries proceed like this:
1 | [INFO] 10.202.37.232:50940 - 51439 "A IN abc.kube-system.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000114128s |
ndots and Its Optimization Problem
The search configuration must be used together with ndots. The default ndots is 1; its function:
if the queried domain contains fewer dots than this value, suffixes from the search domains are tried first.
1 | Resolver queries having fewer than |
A Concrete Example
Suppose our dns configuration is:
1 | search kube-system.svc.cluster.local svc.cluster.local cluster.local |
When we ping abc.123 (this domain has only one dot), the dns server’s log is below. Note the log tries abc.123.kube-system.svc.cluster.local. first, and only tries our domain last.
1 | [INFO] 10.202.37.232:33386 - 36445 "A IN abc.123.kube-system.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.001700129s |
Then when we ping abc.123.def (this domain has two dots), the dns server’s log looks like below. Note the log tries abc.123.def. first:
1 | [INFO] 10.202.37.232:39314 - 794 "A IN abc.123.def. udp 29 false 512" NXDOMAIN qr,rd,ra 104 0.025049846s |
I hope this example makes two points clear:
- No matter what ndots is, the suffixes in the search parameter will be tried in order (our test used a nonexistent domain, so the resolver tried all possibilities)
- An improper ndots setting may pressure the dns server (if the domain exists, the dns query returns as soon as possible without continuing to search, reducing server pressure)
Optimization Discussion
Suppose ndots is now 2 and we want to query baidu.com. Since the dot count 1 is less than the configured 2, suffixes are appended for lookup first:
1 | [INFO] 10.202.37.232:42911 - 55931 "A IN baidu.com.kube-system.svc.cluster.local. udp 57 false 512" NXDOMAIN qr,aa,rd 150 0.000116042s |
Then we produce 3 useless dns query records. For the DNS server, just the domain baidu.com turns into 4x the traffic.
What if n keeps growing — like the default 5 given in Kubernetes? We’d produce even more invalid requests, because not only baidu.com
but even map.baidu.com and m.map.baidu.com would start trying from the search domains — quite a heavy pressure on the DNS server.
My personal suggestions:
- If requests between internal services are very frequent — i.e. we often access domains like
xxx.svc.cluster.local— then keeping ndots larger is fine- But when there are few requests between internal services, I strongly suggest lowering ndots to reduce useless traffic and lighten the dns server’s pressure For my personal use, changing it to 2 is good
Summary
Sorry that most of this article talks about how nameserver is resolved, with little on resolv.conf contents.
The main reason: I’d been reading iptables the previous days, and this happened to come up, so I spent time on it — maybe with a bit of showing off.
When solving problems, understanding the parameters behind them is fairly important. I also pasted some of my experiments, hoping they help everyone — at least understand ndots before considering tuning.