How to take back control of /etc/resolv.conf on Linux (2018)
ctrl.blog
ctrl.blog
Modern programs should use getaddrinfo(3). The glibc implementation of that call will first consult the "hosts:" line of /etc/nsswitch.conf. That file will contain a list to determine how host names are resolved:
The first entry is typically "files". That means consult /etc/hosts file.
Next there could be something to consult avahi/autoip, Ubuntu systems typically do that.
There could also be entries "myhostname" and "mymachines". That's systemd's container support, not discussed here.
If the system uses systemd-resolved there will be an entry "resolve". Than means consult systemd-resolved
Typically the last one is "dns". That's the traditional implementation using /etc/resolv.conf. But if systemd-resolved is configured and working we never get here.
Only if you don't have "resolve" before "dns" in your /etc/nsswitch.conf then /etc/resolv.conf is still relevant.
It appears to me that musl does not use /etc/nsswitch.conf at all, so there /etc/resolv.conf is still relevant.
My only point was if you want "to take control" of your system you first have to look /etc/nsswitch.conf. If "resolve" is before "dns" your /etc/resolv.conf is mostly unused. Any article discussing /etc/resolv.conf without mentioning /etc/nsswitch.conf is very misleading for many users who happen to have systemd-resolved (probably because the distro decided to install and configure it)
Whether nscd is still installed by any distro by default I don't know. I haven't seen it running for years.
It would be fine if overall reliability increased, but it does not. I could write more reliable shell scripts for making WiFi or PPP connections than NetworkManager, which sometimes required a reboot before graciously allowing to connect again. Never happens on the command line.
Thank you for describing the status quo, I'll probably look into Slackware or OpenBSD now.
Fedora and Red Hat and the vast majority of their derivatives use systemd.
OpenSUSE and SUSE and the vast majority of their derivatives use systemd.
In most production environments bar custom distributions used by Google & co. (and those are a minority), systemd is used, presumably with the systemd resolver.
I'm not sure I'd call all users of commercial Linux distributions "laymen" :)
Debian doesn't use nss-resolve(8) by default though. I'd be very surprised if RHEL or any other other distros suitable for a "production environment" were to take that step given systemd-resolved's current level of maturity.
(This is not a knock at systemd-resolved; I use it myself on most of my machines, but unless you want to spend a lot of time debugging weird resolver issues I would not recommend doing so until it has matured a bit more.
I gather your saying systemd-resolved uses DNS? How do I specify the DNS servers used by systemd-resolved? Thanks.
Systemd-resolved has a configuration compiled in. If thats not what you want you can edit /etc/systemd/resolved.conf. It shows what has been compiled in as comments. Actually really editing any systemd configuration or unit file is typically a bad idea. Configuration changes are done preferrably by drop-ins. These are little files to be stored under /etc/systemd/resolved.conf.d/{foo,bar}.conf (Pathnames from memory, typing on my phone, but you should be able to see the naming scheme used. It's quite systematic all over systemd.)
Roughly it works like that (not sure whether distros vary):
primary DNS server address is received from DHCP at runtime
fallback DNS servers are hardwired in the config. I remember at least seeing google 8.8.8.8 there
You can change the primary DNS server not to be set by DHCP, that's probably the first thing may people want to do.
And it is probably a bug because that missing setting seems breaking vpn at least:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1769016
The root cause for "resolve" not being in /etc/nsswitch.conf is libnss-resolve not being installed. It seems to be completely optional with nothing pulling it in.
In my 16.04 systemd-resolved is not even running. That's what I would have expected.
https://github.com/golang/go/issues/22846
Static binaries are fun, but bringing a libc replacement... Can be surprising.
My trouble was with split-horizon networking, and gitlab-runner (which is a nice self-contained go program). But not so nice if your firewall, dns and internal/external ips disagree. Thankfully we could remap some fw rules, so avoided the need for split-horizon host resolution via /etc/hosts (the other alternative being full split-horizon dns).
"files", oh so all the files get checked first, including "/etc/hosts" and "/etc/resolv.conf", right?
No, only "/etc/hosts".
Well why don't we just write it as "/etc/hosts"? Ok, whatever - next up is "myhostname"... wait, my hostname is already in the "/etc/hosts" file, why is it spelled out again?
No no, that refers to systemd's container support (don't ask).
OK... ah, here we go - "resolve", surely that refers to "/etc/resolv.conf".
Nope, that refers to "systemd-resolved".
Well then why didn't they call it that!?!
The culprit is actually NetworkManager. When it detects that systemd-resolved is running it sets the default resolver to systemd-resolved, and automatically makes the override for the user. I think it also does the same for the other supported resolvers (unbound, dnsmasq, etc.).
There is no need to remove or disable systemd-resolved, set chattr +i or anything else (of course, if you're not using you can remove it without problems).
The way packages are like functions in the nix code repo gives you at least fair confidence that things will actually work as intended.
Or at the very least it gives you insight into why it doesn’t. More so compared to a traditional deb/rpm distro in my experience.
Not saying it’s perfect by any means, just that it has worked brilliant for a few small projects of mine involving some pi’s.
At the workplace though, resistance is high amongst my traditionalist RH/centos co-workers. :)
I recently had to change my work's notebook. Since my NixOS configuration was commited in my dotfiles, I simply restored my NixOS configuration and $HOME and everything was working like the old notebook.
Also, bugfix for strange behaviors are written in my NixOS files, so it is like NixOS is my own personal distro.
chattr +i /etc/resolv.conf auditd(8)I'm just a Linux "amateur", but what a huge mess to configure something so trivial/simple... (in Debian/Ubuntu compared to Mac/Windows.)
The problem is that this resolver doesn't understand root queries. That means that `dig +trace` doesn't work anymore, and you have to use an external server for the root zone.
This is such a trivial use case not taken into account by systemd (they don't even want to fix it), that I'm saddened by it..
With 16.04 I get core lockups (Intel microcode) and cannot switch out of X with Ctrl-Alt-F1.
NetworkManager no longer connects the mobile surfstick, so I do it manually now with pppd.
I can't claim that I had a specific problem with systemd, but I never had an issue with SysVinit either.
The overall trend however is that Linux distros accumulate crapware by RedHat's NIH developers. I'm not surprised that NetworkManager is also by RedHat.
Apparently not -- netplan is ubuntu originated. They don't do themselves any favours.
For networking I suggest looking at ConnMan. It has a somewhat unintuitive UI, but it is lightweight, fast, and I have yet to find a wireless network that it will not let me connect to.
You can't switch users within x either, it's been broken since at least 14.04 (according to an old close bug report I found). It works the first time but on subsequent tries you just get the lock screen.
The bug report was closed after several years of nothing.
Since some time switching VTs resets all hardware accelerated applications (i.e. crashes them).
Since some time unlocking the graphical session is randomly broken and requires loginctl session-unlock from another VT (which then promptly triggers bug 2).
These things keep coming and going... it's not even Linux-specific. Stuff just breaks all the time and you shrug and move along, hoping that maybe someday it's fixed. For most of these I wouldn't even know who should receive a bug report. Maybe it's the graphical login, maybe the compositor, maybe mesa, maybe a kernel driver, maybe systemd, maybe maybe...
You could try Ctrl-Alt-F3, F4, etc to get a login terminal.
Not all SysvInit implementations were equal, which was an issue in itself.
It's never been a better time to go BSD instead.
It’s a great distro, gives you loads of choice (even lets you go with systemd) and the developers actually care about open source and the act of giving a shit (unlike what I see from the systemd bug tracker).
Still compiling it, but I'll let you know how it goes
For example "emerge x11-base/xorg-x11" would install X and related utilities, but you wouldn't do that. X is a dependency, what you really want is whatever desktop environment you like. So "emerge qtile" and the rest is handled. The compilation time thing isn't a big issue on any reasonable solid state hardware, and there are precompiled packages available for large popular projects (like office suites).
Install instructions: https://wiki.gentoo.org/wiki/Handbook:AMD64#
Recently I purchased some bluetooth headphones, they don't work well without PulseAudio and so far I haven't gotten anything but raw audio files to playback directly.
Same goes for Firefox. I think there's an intermediate plugin to play directly to ALSA, but for now I'm just playing spotify||netflix via chrome.
Point is, it's slowly becoming harder to avoid Systemd and PulseAudio. Because all the major distros use them, the support, development and documentation for !systemd has died off radically :(
Your point is super valid and it makes me sad, I donate/try to support as much as I can but the amount of developers that are apatheic to niche cases is increasing day by day.
I'm still not sure about the complete garbage thing but sysV init definitely wasn't good enough.
busybox init if all you want are some ttys and fs and openRC are probably what you want.
Stuff like running /bin/true for the non-versioned unit, so that it can coerce the activation to take it sequentially when it actually executes the versioned unit.
There’s also the fact that “no boot is exactly the same” which means reproducing a race condition at boot is basically impossible if it’s anything other than the most trivial.
And anyway. The truth here is that you haven’t actually removed the shell script. You’ve just added boilerplate C-code around it.
If your biggest complaint is/was the composability of shell scripts for the common use-case then this is very much a solved problem in the BSDs (where an init script is usually around 5 lines).
EDIT: please come back with some concrete examples of why I'm wrong. because not only would I like to resolve my own issues; it is also the case that you shouldn't be using downvotes as a "I love systemd so I disagree" button.
I'm still sad that such a good system fell in the hands of Oracle :(
https://devuan.org/os/init-freedom/
The runit init Void Linux is quite slick. I also installed an Artix runit image in a virtual machine that works nicely. We are using it as a supervisor on openSuSE piggybacked onto systemd to run critical infrastructure. It works flawlessly, it's also included in busybox but almost always never enabled.
I tend to avoid systemd so I’ve taken to running Debian machines without it. But if you’re doing something like ephemeral cloud installations then it’s not “worth” the overhead of swapping out your init (and having the consequential reboot and potential instability) so the default is a required target for automation anyway. And if you’ve already set a target then why have another one?
This is why defaults can be harmful if widely deployed and considered non-optional (as systemd is on Debian). They become enforced by convention because they are the only valid common target without wrangling.
For the record, the TC expects maintainers to continue to support
the multiple available init systems in Debian. That includes
merging reasonable contributions, and not reverting existing
support without a compelling reason.I've since switched back to unbound with dnssec-triggerd and life is again glorious.
2. systemd-resolved does not use glibc's resolver.
It is true that systemd-resolved does not understand root queries, and it is unfortunate that the systemd maintainers seem to be unwilling to add support https://github.com/systemd/systemd/issues/5897 . However, this has nothing to do with it using glibc's resolver.
However, as clearly documented in the systemd-resolved(8) man-page: There are 4 supported modes of operation with regard to /etc/resolv.conf, which each have different trade-offs, and are configured by changing /etc/resolve.conf itself:
(1) symlink to /run/systemd/resolve/stub-resolv.conf
(2) symlink to /usr/lib/systemd/resolv.conf
(3) symlink to /run/systemd/resolve/resolv.conf
(4) regular file
`dig +trace` should work properly in modes (3) and (4).
The big difference between those modes is "how do you want programs that don't use the system APIs (and instead do DNS themselves) to behave?". Modes (1) and (2) say "have them send their traffic through systemd-resolved", which as you note has some deficiencies; modes (3) and (4) mean "allow them to talk to the remote DNS servers directly".
Though I don't know why they would "break userspace" like that.
Some programs, like statically linked Go programs, don't use the system APIs because they don't want to link against external libraries. So, if the system is configured to use mDNS for .local hostnames (either via systemd-resolved or via nss_mdns), those programs will mysteriously do the wrong thing, because they _only_ speak plain DNS. You can hack around that, and get them to do the right thing, by having a stub DNS server that they talk to, and have that stub server fire off mDNS requests instead of DNS requests as appropriate. Similar story for things like DNSSEC or DNS-over-TLS. That's what modes (1) and (2) do, they make statically-linked-programs that would otherwise use the system APIs do the right thing.
Other programs, like `dig`, don't use the system APIs because the really do care about speaking DNS directly. They just need the list of DNS servers to talk to and search suffixes to use. Giving them that is what modes (3) and (4) do.
It's literally impossible to keep both sets of programs "doing the right thing" at the same time. So the question is: which group of programs do you want to do the right thing? The systemd devs don't know (much less "know better"), so they made it configurable. Which one is default is up to the distro, not systemd.
You have to use (4) if (and only if) you're going to use a resolvconf(8) implementation other than systemd-resolvconf. For example, I use dhcpcd; for kinda silly reasons dhcpcd and systemd-resolvconf don't work together, so I'm stuck using openresolv as my resolvconf(8) implementation. So I have to use (4). If you're fine using systemd-resolvconf, you should use one of the other 3 modes.
I'm honestly not sure why you would ever use (2) instead of (1). It's like (1), except search paths won't work right for programs that don't use the system APIs. (Maybe it existed before (1) did, and still exists for historical compatibility?)
https://github.com/systemd/systemd/issues/5755
And..."systemd-resolved does not use DNS for local resolution"
https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...
https://zwischenzugs.com/2018/06/18/anatomy-of-a-linux-dns-l...
https://zwischenzugs.com/2018/07/06/anatomy-of-a-linux-dns-l...
https://zwischenzugs.com/2018/08/06/anatomy-of-a-linux-dns-l...
and I haven't even covered caching.
It leads us on a path where daemon authors don't allow us to override the settings to suit us, rather to kept them controlled to suit them.
The arrogance sets a very bad precedent.
I think it helps to think of it as making Linux more like Windows, with all of the plumbing hidden from the user and difficult to tweak by the administrator. "Why would you need to, anyway? We've already taken care of that for you."
However if the only ISP you can get is so hostile, the solution is to tunnel (IPSEC/SSTP/whatever works) all your traffic to a non-hostile network.
That way you dont have to tunnel all your traffic. (Though technically you could also use the tunnel for only DNS, but its not much easier than the solution above if you want this to apply to all your devices)
Yes, it allows you to prevent your ISP manipulating your DNS. Your ISP has no way to know when you are resolving, because it is masked in other HTTPS traffic.
But it also allows the apps to prevent you from manipulating their DNS. You don't know when an app is ignoring the resolver you configured system-wide, because it is masked in its HTTPS traffic.
There is a worrying trend that apps (browsers especially) are ignoring whatever you configured in your system, and are becoming basically a blackbox outside your control with a wide open connectivity to the Internet. No explanation needed, what that means for any privacy left.
(And like, is it really that hard to register your captive portal's name globally, and just let me use a global resolver to get to it? IDK if they're concerned about DNS tunneling, but this happens even on "free" WiFi (where free might mean "watch this ad", which honestly is fine w/ me.))
The people that build the systems that provide these networks likely don't know what DNS tunneling is.
Ideally, the software detecting captive portals on a network and presenting their login page should also use the network-provided DNS server solely to resolve the captive portal domain if it doesn't resolve with the preferred resolver. That would solve this case.
(Also, ideally, captive portals would serve up a URL using DHCP options and not capture traffic.)
That prevents anything, even root, from changing the file, unless chattr -i is called first.
Apparently, they are very file system specific. That's why they are not widely used. I guess a chmod 0444 would not do it, as system tools tend to "fix" wrong permissions automatically (but not permissive attributes).
There is nothing wrong with that but it shouldn't be conflated as choice and paraded as success of such.
There are distributions (e.g. Linux from Scratch) for when the option you want is the NULL option—i.e., doing things yourself. But there are N other managed options, not just one managed option and NULL.
Unfortunately strawman distro is never real and the problems still exist after congratulating the ability to switch to it.
Standard practice for me was to rip out every last shred of NetworkManager on every fresh build, but I didn't realize how many other utilities broke the resolver config as well.
To be fair, in that particular config, resolv.conf never changes as it always points to loopback, with your preferred nameservers only existing in memory in unbound, and in network manager's config.
It's opaque and hard to debug. It may have utility on laptops, but on servers, it is absolutely counterproductive. I could not for the life of me figure out what it was doing w.r.t ipv6 prefix delegation and how it was dealing with dhclient6 internally. My leases would expire but not renew. Eventually, I had to rip it out and create simple configs by hand that work well.
I'll also agree that the benefits on a server are next to non-existant, but at the same time it's just a default that's trivial to turn off, and I could probably count on one hand the number of times it's bitten me while building and deploying tens of thousands of hosts over the last 15 years.
It's a tool I've learned to love on my workstations, and rarely even notice on production systems.
If you're on a RH or Arch derived distro, policy is just the opposite, and if the service is ever magically enabled (aside from Anaconda enabling it after the package was selected at install time), it's a massive bug (I can't find any such bug report in Arch or Fedora).
Not understanding the difference between these is likely part of the problem here.
When something is regularly found to be the problem, and has never made my life any easier, it gets yanked. Sometimes I may have had time to analyze further, other times it may have been an emergency. Regardless, it's been at least 1.5 years since I worked in that environment, and my memory has faded. In the real world, we don't have infinite time to analyze infinite failures.
Here's what I remember:
NetworkManager = problems.
No NetworkManager = no problems.
You may care to look down on people who don't always have the time to analyze every single occurrence of every single problem to the nth degree, and subsequently catalogue the exact cause and fix for reference when posting on forums in the future, and that's fine.
My opinion is, it sucks, and I don't want it.
If you like it, and enjoy its benefits, please do. I'm not denigrating you for your choice.
Lovely. I have between one and two machines that need this. I have 1289 machines that don't.
When I'm on a DHCP network dhclient runs and manages my resolv.conf as expected.
We're still entirely in control in GNU/Linux, if you assert control. It turns out however that most people actually want the distros to do all the magic and for things to Just Work.
Turns out it's an 'upstart' type ubuntu crapfest that's unlikely to survive as long as 20.04.
I wouldn't mind, but the very first machine I tried to configure with it was an ntop machine, I wanted 2 interfaces to come up with no IPs attached. Turns out it's not possible.
Debian prides itself on stability but voted for an 'init' that is rapidly changing with unlimited scope outside the purview of Debian. Thus we have a lot of extremely questionable networking functionality like 'predictable network names' and more just rolled out and 'imposed' on everyone without discussion, debate or technical scrutiny by a single group.
Alpine Linux servers and VMs boot in seconds and are easy to configure and use. Void is equally fast for desktop users. The real problem seems to be a lot of developers are unfamiliar with the shell and its extensive use in everything from the system to the apps ecosystem. And we are left with a situation where the shell is demonized by people who are not familiar with it.
I don't feel strongly about networkd or NetworkManager either way, but I am curious on the meta-debate. I am personally a fan of the modernization of the linux userspace in the last 10 years (systemd, wayland, networkmanager), but I also understand that the puzzle pieces don't always fit together. However the argument against new things always seems to come down to "not what I am used to".
Other than tradition what is wrong with explicitly configuring NetworkManager to configure /etc/resolv.conf "my way" instead of going behind it's back. Does it not have a "DNS Server: manual" option?
> However the argument against new things always seems to come down to "not what I am used to".
For me it's always: I can't debug this. I remember installing RHEL 7.0 just after release and nmtui crashing on me with some weird problem in glib when setting a hostname using it. I also remember trying to debug openvpn in network manager (https://i.imgur.com/zbaRcST.mp4) and both of these are beaten paths...
I'm able to somehow get around most shell scripts but when something crashes in C land on a customer box - what a PITA! It's not impossible but the amount of work I have to invest is just sooo huge. I remember yelling at reddit about some gnome stuff - and I've got a link to several powerpoint presentations... give me technical documentation not powerpoints and give me tools to troubleshoot that shit. No strace should be a last resort and not your goto tool: https://blog.uberspace.de/systemd-error-getting-authority-ko... (link in german, but google translate should work)
> this is dishonest
Of course it is, I was trying to trigger a response :)
Most of the other ones just say "stuff is broken too bad :(" when a connection fails. wpa_supplicant tells you everything that's wrong and not only that, usually gives you an easy way to fix it.
Being an OS for nerds, I'm surprised NetworkManager doesn't handle for manual updates to resolv.conf
The simple solution is, make your manual overrides through NetworkManager, or don't use it.
Edit: now I notice a sibling comment already asked almost the same question, but it was greyed out.
chattr +i /etc/resolv.conf
ie. sensible default for many people, but super easy to override
the suspicion that a user is so incapable of having already set DNS servers is so strong that whatever-is-doing-this doesn't even bother to ask if or check whether the resolv.conf file is different from default.
this seems a little too critical of, possibly insulting to the intelligence of, people who use these versions of linux.
Yes well, that's what happens when package managers, with their dependence on maintainers and whatnot, are your culturally lauded software distribution method.