If one DNS server is down, use the cached result or another server.
DNS is some of the most distributable, cachable data I can imagine.
If one DNS server is down, use the cached result or another server.
DNS is some of the most distributable, cachable data I can imagine.
Of course small DNS providers will find it hard to run a system this way, but the larger providers follow the same architecture - anycast and multiple servers at each location. Google and OpenDNS for a start use this pattern - the famous 8.8.8.8 and 8.8.4.4 are in fact multiple server clusters all around the world.
This is a simple case of a DNS provider evidently not having their crap together when it comes to a DDoS of any decent size. There are plenty of other DNS providers out there who are working fine currently, this DDoS is limited to just the one. It's all of that company's clients that are impacted.
I'm sure that DynETC will post something afterwards about how this was the largest DDoS they've ever encountered by many orders of magnitude and that there was no way for anyone to ever be prepared to take so much traffic... but at this point I think they're inept.
edit: just came back up. The TTLs on a few common records explain why this was so obviously a problem...
- www.reddit.com: 300 seconds (pointer to a fastly.net address, which is hosted via dynect)
- prod.reddit.map.fastlylb.net: 30 seconds
- api.twitter.com: 300 seconds
- herokussl.com: 3600 seconds
So, an outage of five minutes is enough to take out access to all of reddit and twitter. An outage of an hour is enough to take out heroku. Of course, these are best case scenarios: in reality, 50% of your users would lose access to reddit after 15 seconds (the fastly record), and so on.
Indeed. How can it be so broken? And, it seem that it would make sense to use the last known value if upstream is unavailable.