It is always the DNS
ruben.is.verweg.com
ruben.is.verweg.com
The bug here as I understand it: this hosting provider has a client whose DNS authority server DNSSEC-signs their records. That server is misconfigured.
When you look up a nonexistent record in DNS, you get a simple response back that says there aren't any matching records. Easy enough!
But you can't do that in DNSSEC, because denial "needs" to be signed. What DNSSEC does instead is essentially return a descriptor of a lexicographically ordered span of records that does exist, and doesn't include the record you were looking for. You notice that, and interpret as denial. See, simple! Oh, also, the names in those records are password-hashed, because otherwise DNSSEC would be publishing raw zone dumps to serve errors. Except those password hashes are very straightforwardly cracked. But I digress. The records carrying these spans in modern servers are called NSEC3s. Think of a tuple (start-name, stop-name, record-types).
If you asked this client's server for a nonexistent A record, you'd get an NSEC3 back that properly established the covering span of names in which A was nonexistent --- but it erroneously signaled that other record types for that span (most notably MX) were also nonexistent.†
The fun part of this bug is that it's effectively a vulnerability: an attacker can query random recursive caches for nonexistent record types to generate a cached denial records for records that do exist. Which is not something that can't go wrong in ordinary unsigned DNS! It's a vulnerability that DNSSEC adds to the DNS.
This is your periodic reminder that virtually none of the industry's largest tech companies (and concomitantly well-resourced security teams) use DNSSEC, which you can confirm quickly for yourself by running `host -t ds <domain>` and noting that they don't have DS records.
Previous (recent) thread, in which DNSSEC blew up (checks notes) all of Slack: https://news.ycombinator.com/item?id=29378633
† (If I've got the particulars of this wrong, that makes it even funnier, since it's suggesting that it's difficult even to describe what can go wrong with DNSSEC).
- never trust a single DNS server as a source of truth
- always rely on transport encryption, which DNS on port 53 (via UDP) does not offer
- verify your results statistically, rotate servers (DNS ronin), and endpoint protocols in order to verify there's no MITM
- deactivate message compression if possible. Naive implementations are vulnerable to both stack and buffer overflows (see NAMEWRECK).
- don't rely on DNS cookies, and always assume there can be multiple records with different types in a reply to a single question.
- DNS TTL 0 is a fix for stupidity, and nothing else.
If any of the above isn't respected it's your fault as a DNS client, because the protocol was designed when the internet was a nice place and everybody literally knew everybody via the /etc/hosts file dumps.
That's why I'm a strong believer in DNS via HTTPS, because having a better and more reliable transport protocol would fix a lot of the issues people/clients have to face.
DNS via TLS turned out to be useless because the dedicated port 853 for it literally gets blocked by ISPs and nobody can do shit about it.
PowerDNS tried this early on, it ended poorly because a client with a large installed base assumed that answers following the question section would start with a compression pointer.
There have been a few DNS flags days in recent years where recursive DNS resolvers would remove certain work arounds for bugs in authoritative DNS servers (or server configs).
In short, classical DNS has a long list of issues.
As for caching the NSEC or NSEC3, that's an optimization that is not possible without DNSSEC, but DNS resolution would work equally well without this feature.
Most recursive resolvers are not open to the public, so they don't need extensive protection from malicious clients. Caching an NSEC or NSEC3 helps against certain denial of service attacks. But it has a cost in term of failing in the case of bad NSEC/NSEC3 data.
That said, point of DNS flags days is to just break protocol violations instead of creating a work around. If you don't, you get into the mess where certain websites only work with certain browsers, etc. Many years ago, browsers would take a very lenient stance at wrong web PKI certificates allowing the user to grant an exceptions. These days, that is mostly over.
This would be an excellent security research topic. How to design non-single point of failure security that both fail safe and do not cause down time when misconfigured.
This question lies in the same category of questions which I’ve never managed to get answers to. Why fetching email over https is instant, but over imap it may take minutes to “sync” (or what it does there)? Why browsing is instant, but ftp was dead slow? Why all the formats (mailbox, uue, etc), when you could have a proper database with column encodings? I’m not asking anyone to answer these, cause it would be too offtopic, but these arcane issues have never left me alone.
I understand that most of it is historical, but what the heck, internet.
That's what I used to think. But somehow in practice, at least for me, using DoH seems to return responses much faster (DoH response takes about ~1/10th the time). Don't know why... Maybe UDP packets are being dropped in bursty traffic by the ISP?
Nobody really knows why that is the case. Maybe some ISPs don't spend as much time optimizing UDP traffic compared to TCP.
Maybe your router or ISP is messing with UDP or intercepting DNS and forcing you to use their DNS servers. Lots of “firewall” middleboxes do that by default. Or maybe there’s an path MTU issue that triggers retransmit on every UDP packet, but not on an established TCP connection that has learned the correct MTU.
The DNS JSON API is a google only thing, because even cloudflare's DNS use different MIME types and don't support the complete API and have varying support for base64 with/without padding characters and the full 65k frame size that the spec and drafts define. EDNS request options are also mostly unsupported in practice.
Even multiple questions (A/AAAA) are partially supported and oftentimes fail dependent on which server you get as an endpoint. The URL schema is also different from server to server because everybody implements their own thing.
Last year my company decided to move one of its public services to a new hosted vendor and change their domains entirely. This would involve (among other things) changing the domain that customers were using and changing live traffic to point from one vendor to another. Of course nobody budgeted for any downtime, and the system was so little-known that it would have taken forever to figure out what all would be affected by a maintenance window. A less experienced engineer might consider this trivial; just change the DNS records and flip from one live system to another. Easy, right?
Luckily, something like 15 years before, I had managed (among other things) DNS records for a large internet site. Because we were essentially hand-crafting Bind zone files and making all sorts of weird changes to tons of records, I learned all about the DNS ecosystem. I learned about all the different layers of the stack, all the different forms of caching, the common problems (UDP vs TCP, EDNS0, firewalls, Apex record limitations, negative cache, ignored TTL), spam features, service identifiers, XFERs, and the fact that there's a dozen different RFCs all laid on top of each other to form "the DNS protocol".
Keeping in mind all the potential problems, I was able to plan our migration steps thoroughly enough for the 144-hour-long change window. I knew to prepare my SOA, NS, A & CNAME changes, verify the records on the nameservers (and from client devices) pre-change, wait 2 days after changing the TTL limit, verify with resolvers around the world to confirm everything was picked up, and validate on each device that the change worked correctly. The process worked beautifully and the customers never noticed a thing. And if there had been a problem, my changes would have resulted in no more than about 60 seconds downtime.
What struck me at the end wasn't how much bizarrely complex and deep knowledge was required just to make the switch-over successful. It was that I had learned everything I needed to know 15 years before, and all that technical knowledge remained the same. I think that's only happened one or two other times in my career (one related to TCP weirdness, another related to C programming)