In Anatomy of a Linux DNS Lookup – Part I I covered:
and in Anatomy of a Linux DNS Lookup – Part II I covered:
and ended up here:
A (roughly) accurate map of what’s going on
Unfortunately, that’s not the end of the story. There’s still more things that can get involved. In Part III, I’m going to cover NetworkManager and dnsmasq and briefly show how they play a part.
Other posts in the series:
As mentioned in Part II, we are now well away from POSIX standards and into Linux distribution-specific areas of DNS resolution management.
In my preferred distribution (Ubuntu), there is a service that’s available and often installed for me as a dependency of some other package I install called NetworkManager. It’s actually a service developed by RedHat in 2004 to help manage network interfaces for you.
What does this have to do with DNS? Install it to find out:
$ apt-get install -y network-manager
In my distribution, I get a config file.
$ cat /etc/NetworkManager/NetworkManager.conf [main] plugins=ifupdown,keyfile,ofono dns=dnsmasq [ifupdown] managed=false
dns=dnsmasq there? That means that NetworkManager will use
dnsmasq to manage DNS on the host.
The dnsmasq program is that now-familiar thing: yet another level of indirection for
Technically, dnsmasq can do a few things, but is primarily it acts as a DNS server that can cache requests to other DNS servers. It runs on port 53 (the standard DNS port), on all local network interfaces.
So where is
dnsmasq running? NetworkManager is running:
$ ps -ef | grep NetworkManager root 15048 1 0 16:39 ? 00:00:00 /usr/sbin/NetworkManager --no-daemon
dnsmasq process exists:
$ ps -ef | grep dnsmasq $
Although it’s configured to be used, confusingly it’s not actually installed! So you’re going to install it.
Before you install it though, let’s check the state of
$ cat /etc/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 10.0.2.2 search home
It’s not been changed by NetworkManager.
dnsmasq is installed:
$ apt-get install -y dnsmasq
dnsmasq is up and running:
$ ps -ef | grep dnsmasq dnsmasq 15286 1 0 16:54 ? 00:00:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor=.,19036,8,2,49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5
/etc/resolv.conf has changed again!
root@linuxdns1:~# cat /etc/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 127.0.0.1 search home
dnsmasq is serving on all interfaces at port 53:
$ netstat -nlp4 Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 15286/dnsmasq tcp 0 0 10.0.2.15:53 0.0.0.0:* LISTEN 15286/dnsmasq tcp 0 0 172.28.128.11:53 0.0.0.0:* LISTEN 15286/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1237/sshd udp 0 0 127.0.0.1:53 0.0.0.0:* 15286/dnsmasq udp 0 0 10.0.2.15:53 0.0.0.0:* 15286/dnsmasq udp 0 0 172.28.128.11:53 0.0.0.0:* 15286/dnsmasq udp 0 0 0.0.0.0:68 0.0.0.0:* 10758/dhclient udp 0 0 0.0.0.0:68 0.0.0.0:* 10530/dhclient udp 0 0 0.0.0.0:68 0.0.0.0:* 10185/dhclient
3) Unpicking dnsmasq
Now we are in a situation where all DNS queries are going to
127.0.0.1:53 and from there what happens?
We can get a clue from looking again at the
/var/run folder. The
resolvconf has been changed to point to where
dnsmasq is being served:
$ cat /var/run/resolvconf/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 127.0.0.1 search home
while there’s a new
dnsmasq folder with its own
$ cat /run/dnsmasq/resolv.conf nameserver 10.0.2.2
which has the nameserver given to us by
We can reason about this without looking too deeply, but what if we really want to know what’s going on?
4) Debugging Dnsmasq
Frequently I’ve found myself wondering what dnsmasq’s state is. Fortunately, you can get a good amount of information out of it if you set change this line in
Now, if you do a simple:
$ ping -c1 bbc.co.uk
you will see something like this in
[...] indicates that the line’s start is the same as the previous one):
Jul 3 19:56:07 ubuntu-xenial dnsmasq: query[A] bbc.co.uk from 127.0.0.1 [...] forwarded bbc.co.uk to 10.0.2.2 [...] reply bbc.co.uk is 18.104.22.168 [...] reply bbc.co.uk is 22.214.171.124 [...] reply bbc.co.uk is 126.96.36.199 [...] reply bbc.co.uk is 188.8.131.52 [...] query[PTR] 184.108.40.206.in-addr.arpa from 127.0.0.1 [...] forwarded 220.127.116.11.in-addr.arpa to 10.0.2.2 [...] reply 18.104.22.168 is NXDOMAIN
which shows what
dnsmasq received, where the query was forwarded to, and what reply was received.
If the query is returned from the cache (or, more exactly, the local ‘time-to-live’ for the query has not expired), then it looks like this in the logs:
[...] query[A] bbc.co.uk from 127.0.0.1 [...] cached bbc.co.uk is 22.214.171.124 [...] cached bbc.co.uk is 126.96.36.199 [...] cached bbc.co.uk is 188.8.131.52 [...] cached bbc.co.uk is 184.108.40.206 [...] query[PTR] 220.127.116.11.in-addr.arpa from 127.0.0.1
and if you ever want to know what’s in your cache, you can provoke dnsmasq into sending it to the same log file by sending the
USR1 signal to the dnsmasq process id:
$ kill -SIGUSR1 <(cat /run/dnsmasq/dnsmasq.pid)
and the output of the dump looks like this:
Jul 3 15:08:08 ubuntu-xenial dnsmasq: time 1530630488 [...] cache size 150, 0/5 cache insertions re-used unexpired cache entries. [...] queries forwarded 2, queries answered locally 0 [...] queries for authoritative zones 0 [...] server 10.0.2.2#53: queries sent 2, retried or failed 0 [...] Host Address Flags Expires [...] linuxdns1 172.28.128.8 4FRI H [...] ip6-localhost ::1 6FRI H [...] ip6-allhosts ff02::3 6FRI H [...] ip6-localnet fe00:: 6FRI H [...] ip6-mcastprefix ff00:: 6FRI H [...] ip6-loopback : 6F I H [...] ip6-allnodes ff02: 6FRI H [...] bbc.co.uk 18.104.22.168 4F Tue Jul 3 15:11:41 2018 [...] bbc.co.uk 22.214.171.124 4F Tue Jul 3 15:11:41 2018 [...] bbc.co.uk 126.96.36.199 4F Tue Jul 3 15:11:41 2018 [...] bbc.co.uk 188.8.131.52 4F Tue Jul 3 15:11:41 2018 [...] 184.108.40.206 4 R NX Tue Jul 3 15:34:17 2018 [...] localhost 127.0.0.1 4FRI H [...] <Root> 19036 8 2 SF I [...] ip6-allrouters ff02::2 6FRI H
In the above output, I believe (but don’t know, and ‘?’ indicates a relatively wild guess on my part) that:
- ‘4’ means IPv4
- ‘6’ means IPv6
- ‘H’ means address was read from an
- ‘I’ ? ‘Immortal’ DNS value? (ie no time-to-live value?)
- ‘F’ ?
- ‘R’ ?
Alternatives to dnsmasq
dnsmasq is not the only option that can be passed to dns in NetworkManager. There’s
none which does nothing to
default, which claims to ‘update
resolv.conf to reflect currently active connections’, and
unbound, which communicates with the
unbound service and
dnssec-triggerd, which is concerned with DNS security and is not covered here.
End of Part III
That’s the end of Part III, where we covered the NetworkManager service, and its
Let’s briefly list some of the things we’ve come across so far:
Or you might like Docker in Practice