* http://jdebp.eu./FGA/dns-client-all-proxies-must-provide-sam...
There's a lot of such misunderstanding here in this discussion, with people asserting, erroneously, that there's a standard, or at least conventional, behaviour for multiple proxy DNS server listings in /etc/resolv.conf, that systemd-resolved is deviating from. In fact, there is a very prominent difference between two widely used softwares that predates systemd by decades, namely nslookup and the C library's DNS client. In actual fact there is not one single conventional behaviour. This particular Frequently Given Answer touching upon the differences in the access patterns dates from 2001, and people are still making this conceptual error about how DNS works 16 years later.
There is absolutely no guarantee that dns resolvers will be queried in order. The implementations of this vary drastically - some implementations make the request to all dns resolvers configured and simply use the response that comes back the fastest.
The implemented a hack that seemed to "work" but made poor assumptions as to why. This is not the correct way to implement split horizon DNS.
SystemD is following a standard, while people are relying on an implementation. One that has been stable for a long time and has become a de-facto standard. Think of e.g. memcpy on BSD - even though the standard says memcpy between overlapped regions is not defined, the implementation does memmove in that case, because that's what people expect to happen. If you're replacing a system that has historically provided guarantees above and beyond those given by a bespoke standard, you should emulate them, or you'll cause users a lot of pain.
SystemD devs have chosen to inflict pain on their users, hence the users are not happy.
There was at least a "defacto" standard in that many Linux distribution resolv.conf man pages said things like "If there are multiple servers, the resolver library queries them in the order listed." (verbatim quote from some Linux distribution...I concede though, that different distributions may not have spelled that out)
Once one of these is received, it's been my expectation that the query should been marked as failed, and not reattempted.
Here's a list of all RFCs related to DNS: https://www.isc.org/community/rfcs/dns/
In https://tools.ietf.org/html/rfc1034 this is discussed:
The resolver always starts with a list of server names to query (SLIST).
This list will be all NS RRs which correspond to the nearest ancestor
zone that the resolver knows about. To avoid startup problems, the
resolver should have a set of default servers which it will ask should
it have no current NS RRs which are appropriate. The resolver then adds
to SLIST all of the known addresses for the name servers, and may start
parallel requests to acquire the addresses of the servers when the
resolver has the name, but no addresses, for the name servers.
BIND9 supports 'views' (iirc) which allow for splitting queries between internal vs. external DNS. But that would be the resolver as opposed the users desktop implementing that.
"The strategy is to cycle around all of the addresses for all of the servers with a timeout between each transmission."
Note that it does not specify an order -- the resolver determines that, and RFC 1035 even recommends keeping a weighted list of nameservers prioritized by response time:
"To complete initialization of SLIST, the resolver attaches whatever history information it has to the each address in SLIST. This will usually consist of some sort of weighted averages for the response time of the address, and the batting average of the address (i.e., how often the address responded at all to the request)."
...which is exactly what systemd is doing. The glibc behavior is a naive simplest-implementation approach, and if your infrastructure depends on an implementation detail of a specific libc resolver, you should install proper configurable resolvers and use those instead.
The fact that all nameservers are expected to return identical results is inherent in the design of DNS. It's one namespace, and if you need to split it, the client resolver is the wrong place to do that too.