Hacker News Reader

Comment by diafygi | Hacker News Reader

I'm a little confused about the mechanics of how dns failover works. Would you care to explain a bit more?

There is a few things possible here. You can do global load balancing which will serve a DNS entry for a hostname based on business rules. Those business rules can contain things like geo targeting, and/or they can also contain health checks. So you could direct traffic via dns based on if a region is online at all, or more granular things like is it meeting performance criteria, etc.

Even more simple, is DNS round robin (just putting more than one entry in the reply for a dns request). So in terms of failover, if one of these servers in the reply stops working, generally another one is, and browsers and other devices generally understand that they can try all addresses listed for a given host.

Isn't DNS cached at some level on the users machines such that if a particular hostname went down clients which cached that hostname will still be directed to it?

FWIW I'm making it a thing today to read up on DNS as this post made me woefully aware of how little I understand it. So feel free to skip this question and I'll probably answer it myself tonight.

Typically, with round robin DNS, the browser will try the other records if one fails. Its not perfect, but its good enough.

In a perfect world, you use short TTLs (30-60 seconds), and withdraw a record from being advertised as soon as your health checks are failing for it.

That makes me think of something like an API-hungry mobile application that might make ~5 requests a second. If it had a 60-second cached DNS record to a host name that corresponded to a failed host that would lead to 300 failed requests and a really shitty user experience :(.

Your correct, but this is combatted in practice through UX usually. Doing work in the background, doing retry w/ decay, bubbling up things that didn't go well to the user at the right time, etc. For a client to just spam 300 requests the moment there is a hiccup that is generally not good. At scale its indistinguishable from a DDOS (been there, hah!).

Also for it to be that jarring we are talking about a failure at the exact moment you are using the data-heavy app, and not 1 minute before. Every time you go to use the app its likely the dns is re-queried.

So, at the end of the day if you have a mission critical thing running where you need to try very hard and can't tolerate 60 seconds of failure, you would bake multiple paths to a backend in the client configuration at some point. So the client would know if backend A is down it can do client side failover to backend B.

Sometimes this technique is used in conjunction with a highly fault tolerant server stuff, and not as a replacement for it.

Thanks for the follow-up, it's always insightful to hear from people who have seen these things in the trenches on larger systems then I've had privy to.

Reply on news.ycombinator.com