Hacker News Reader

Comment by pm2222 | Hacker News Reader

I'm confused by this. Are the DNS servers inside the backbone, or outside?

  To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation,  making these locations declare themselves unhealthy and withdraw those BGP advertisements. The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers.

View on HN

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world through the PoPs only)

The DNS servers run at the PoPs, but only BGP advertise the DNS addresses when the PoP determines it's healthy. If there's no connectivity back to a FB datacenter (or perhaps, no connectivity to the preferred datacenter), the PoP is unhealthy and won't advertise the DNS addresses over BGP.

Since the BGP change that was pushed eliminated the backbone connectivity, none of the PoPs were able to connect to datacenters, and so they all, independently, stopped advertising the DNS addresses.

So that's why DNS went down. Of course, since client access goes through load balancers at the PoPs, and the PoPs couldn't access the datacenters where requests are actually processed, DNS being down wasn't a meaningful impediment to accessing the services. Apparently, it was an issue with management (among other issues).

Disclosure: I worked at WhatsApp until 2019, and saw some of the network diagrams. Network design may have changed a bit in the last 2 years, but probably not too much.

/* just observing the data presented in the Cloudflare article (https://blog.cloudflare.com/october-2021-facebook-outage/) and disagreeing with the conclusion :-)

While 129.134.30.0/23 (subnet where and a and b nameservers reside) has indeed been withdrawn (according to FB postmortem by DNS automation tooling, 129.134.0.0/17 that is the shorter prefix (perhaps summary at the edge) was still present, however, didn't have longer prefixes (e.g 129.134.30.0/24 and 129.134.31.0/24 we normally see anycasted externally) internally. In other words - routing towards FB DNS subnet (I haven't looked into 185.89.218.0/23 which is where 2 other authoritative nameservers reside) still worked up to the FB border, the traffic was dropped (routed to Null) by FB edge, since it didn't have more specifics internally.

This, combined with TTL of 60 seconds led to almost immediate global DNS failure and all other stuff you have been reading about.

That particular subnet has a covering prefix, but I don't think the other two DNS subnets do, and I had checked on the WhatsApp authoritative subnets, because I have greater affinity for WhatsApp. The WhatsApp subnets don't usually have a covering prefix (and I did check a looking glass during the outage and there were no announcements visible at least at that point).

For those with a covering prefix, the diagnosis is a little bit different as you said, traffic would still flow to whichever FB PoPs advertise the covering prefix, but then it loops in FB, because the PoP doesn't know where to send it, since nowhere was advertising the specific /24. As opposed to the addresses with zero announcements, where the traffic doesn't make it to FB, but gets dropped somewhere else.

Ok so the DNS servers at PoPs, outside of backbone, did not go down.

Does it mean they can respond with public IPs meaningful for local PoP only and are not able to respond with IPs as directions to other PoPs or FB's main DCs? So that has to mean different public IPs are handed out at different PoPs, right?

I'm not quite sure I understand the question exactly, but let me give it a try.

So, first off, each pop has a /24, so like the seattle-1 pop which is near me has 157.240.3.X addresses; for me, whatsapp.net currently resolves to 157.240.3.54 in the seattle-1 pop. these addresses are used as unicast meaning they go to one place only, and they're dedicated for seattle-1 (until FB moves them around). But there are also anycast /24s, like 69.171.250.0/24, where 69.171.250.60 is a loadbalancer IP that does the same job as 157.240.3.54, but multiple PoPs advertise 69.171.250.0/24; it's served from seattle-1 for me, but probably something else for you unless you're nearby.

The DNS server IPs are also anycast, so if a PoP is healthy, it will BGP advertise the DNS server IPs (or at least some of them; if I ping {a-d}.ns.whatsapp.net, I see 4 different ping times, so I can tell seattle-1 is only advertising d.ns.whatsapp.net right now, and if I worked a little harder, I could probably figure out the other PoPs).

Ok, so then I think your question is, if my DNS request for whatsapp.net makes it to the seattle1 PoP, will it only respond with a seattle-1 IP? That's one way to do it, but it's not necessarily the best way. Since my DNS requests could make it to any PoP, sending back an answer that points at that PoP may not be the best place to send me.

Ideally, you want to send back an answer that is network local to the requester and also not a PoP that is overloaded. Every fancy DNS server does it a little different, but more or less you're integrating a bunch of information that links resolver IP to network location as well as capacity information and doing the best you can. Sometimes that would be sending users to anycast which should end up network local (but doesn't always), sometimes it's sending them to a specific pop you think is local, sometimes it's sending them to another pop because the usual best pop has some issue (overloaded on CPU, network congestion to the datacenters, network congestion on peering/transit, utility power issue, incoming weather event, fiber cut or upcoming fiber maintenance, etc).

But in short, different DNS requests will get different answers. If you've got a few minutes, run these commands to see the range of answers you could get for the same query:

    host whatsapp.net # using your system resolver settings
    host whatsapp.net a.ns.whatsapp.net # direct to authoritative A
    host whatsapp.net b.ns.whatsapp.net # direct to B
    host whatsapp.net 8.8.8.8 # google public DNS
    host whatsapp.net 1.1.1.1 # cloudflare public DNS
    host whatsapp.net 4.2.2.1 # level 3 not entirely public DNS
    host whatsapp.net 208.67.222.222 # OpenDNS
    host whatsapp.net 9.9.9.9 # Quad9

You should see a bunch of different addresses for the same service. FB hostnames do similar things of course.

Adding on, the BGP announcments for the unicast /24s of the PoPs didn't go down during yesterday's outage. If you had any of the pop specific IPs for whatsapp.net, you could still use http://whatsapp.net (or https://whatsapp.net ), because the configuration for that hostname is so simple, it's served from the PoPs without going to the datacenters (it just sets some HSTS headers and redirects to www.whatsapp.com, which perhaps despite appearances is a page that is served from the datacenters and so would not have worked during the outage).

  Ok, so then I think your question is, if my DNS request for whatsapp.net makes it to the seattle1 PoP, will it only respond with a seattle-1 IP? That's one way to do it, but it's not necessarily the best way. Since my DNS requests could make it to any PoP, sending back an answer that points at that PoP may not be the best place to send me.

  Ideally, you want to send back an answer that is network local to the requester and also not a PoP that is overloaded. Every fancy DNS server does it a little different, but more or less you're integrating a bunch of information that links resolver IP to network location as well as capacity information and doing the best you can. Sometimes that would be sending users to anycast which should end up network local (but doesn't always), sometimes it's sending them to a specific pop you think is local, sometimes it's sending them to another pop because the usual best pop has some issue (overloaded on CPU, network congestion to the datacenters, network congestion on peering/transit, utility power issue, incoming weather event, fiber cut or upcoming fiber maintenance, etc).

Right I was hoping the DNSs of FB ought to be smarter than usual and let's say when DNS at Seattle-1 cannot reach backbone it'd respond with IP of perhaps NYC/SF before it starts the BGP withdrawal.

Thanks for the write up and I enjoy it.

> Right I was hoping the DNSs of FB ought to be smarter than usual and let's say when DNS at Seattle-1 cannot reach backbone it'd respond with IP of perhaps NYC/SF before it starts the BGP withdrawal.

The problem there is coordination. The PoPs don't generally communicate amongst themselves (and may not have been able to after the FB backbone was broken, although technically, they could have through transit connectivity, it may not be configured to work that way), so when a PoP loses its connection to the FB datacenters, it also loses its source of what PoPs are available and healthy. I think this is likely a classic distributed systems problem; the desired behavior when an individual node becomes unhealthy is different than when all nodes become unhealthy, but the nature of distributed systems is that a node can't tell if its the only unhealthy node or all nodes became unhealthy together. Each individual PoP did the right thing by dropping out of the anycast, but because they all did it, it was the wrong thing.

You are to the point and precise. This is exactly the problem.

  Each individual PoP did the right thing by dropping out of the anycast, but because they all did it, it was the wrong thing.

Somehow I feel the design is flawed because if abuses DNS server status a bit. I mean DNS server down and BGP withdrawal for the DNS server is a perfect combination, however connectivity between DNS and backend server down, DNS up and BGP withdrawal for DNS server is not. DNS did not fail and DNS should just fall back to some other operational DNS perhaps a regional/global default one.

I think this is not necessarily a flaw of the design. It's a fundamental weakness of the real world.

Either you can take the backbone being unavailable as a symbol that the PoP is broken, and kill the PoP; or you can take the backbone being unavailable as a symbol that the backbone is broken and do your best.

When either interpretation is wrong, you'll need humans to come around and intervene. It's much more common that the only the PoP is broken, so having that require intervention results in more effort.

The flaw here is more that the intervention required to get the backbone back was hard to do because internal tools to bring back the backbone relied on DNS which relied on the backbone being up. As well, there were some reports that physical security relied on the backbone being up. And that restoring the backbone needed physical access.

This isn't the first largescale FB outage where the root cause was a bad configuration was pushed globally quickly. It's really something they need to learn not to do. But, even without that, being able to get key things running again, like the backbone, and DNS, and the configuration system(s), and centralized authentication, needs to be doable without those key systems running. I suspect at least some of that will be improved on, and hopefully regularly practiced.

I'm not going to claim to be a BGP expert, but as I understand it the way BGP propagation tends to work makes it a pretty global thing just in terms of how the router hardware handles stuff, which makes it unusually tricky to avoid.

I don't disagree about the general problem mind, I just have a feeling that fixing "don't push configs globally" for BGP specifically is unusually complicated.

  internal tools to bring back the backbone relied on DNS which relied on the backbone being up

So are you referring to same DNS servers sitting outside the backbone at the various PoPs? I'd imagine some internal DNS servers which stays in the backbone at use here, unless of course the FB engineers themselves were disconnected from those internal DNS servers.

I don't recall how internal DNS was setup (and determining from the outside isn't really possible), but there were comments in the incident report that DNS being unavailable made it harder to recover.

https://medium.com/@jag_tsm/facebooks-recent-outage-explaine...

Thanks for the very helpful explanation!

If I understand it correctly they have DNS servers spread out at different locations. These locations are also BGP peering locations. If the DNS server at a location cannot reach the other datacenters via the backbone it stops advertising their IP prefixes. The hope is that traffic will then instead get routed to some other facebook location that is still operational.

The wording is a bit unclear (rushed, no doubt) but I expect this means the DNS servers stopped announcing themselves as possible targets for the anycasted IPs Facebook uses for its authoritative DNS [1], since they learned that the network was deemed unhealthy. If they all do that nobody will answer traffic sent to the authoritative DNS IPs and nothing works.

[1] See "our authoritative name servers that occupy well known IP addresses themselves" mentioned earlier

The anycasted IPs for DNS servers make sense to me and the BGP withdrawal too when the common cases are perhaps one or a few PoPs lost connectivity to the backbone/DC rarely every one of them fails at the time.

I was hoping perhaps the DNS servers at PoPs can be improved by responding with public IPs for other PoPs/DCs and only when that is not available start the BGP withdrawal. Or I can presume the available DNSs at PoPs decrease over time and the remaining ones getting more and more requests until finally every one of them is cut off from the internet.

Facebook’s authoritative DNS servers are at the borders between Facebook’s backbone and the rest of the Internet.

Reply on news.ycombinator.com