Networking is so fragile. I feel like DNS failover to another AS in another datacenter is the only solution for web services to actually have resilience against such failures.
I've wondered if it's simply because I don't know it well enough or if we need an easier, more simple solution to replace DNS in internal networks. There are a ton of ways to do service discovery.
IMO, and emphatically not ripping on you: it's this one. DNS is a pretty simple protocol and at "you don't have ten thousand developers" scale managing it is pretty easy. It might take you a while the first time, for sure--but it won't the second, third, or twelfth time. =)
In particular, even setting aside explicit failover cases (which do require some outside monitoring and staging to make useful, i.e. when do you failover?), round-robin DNS actually does a lot to provide redundancy, internal to a single data center or deployment, when coupled with SRV records.
Even more simple, is DNS round robin (just putting more than one entry in the reply for a dns request). So in terms of failover, if one of these servers in the reply stops working, generally another one is, and browsers and other devices generally understand that they can try all addresses listed for a given host.
FWIW I'm making it a thing today to read up on DNS as this post made me woefully aware of how little I understand it. So feel free to skip this question and I'll probably answer it myself tonight.
In a perfect world, you use short TTLs (30-60 seconds), and withdraw a record from being advertised as soon as your health checks are failing for it.
Also for it to be that jarring we are talking about a failure at the exact moment you are using the data-heavy app, and not 1 minute before. Every time you go to use the app its likely the dns is re-queried.
So, at the end of the day if you have a mission critical thing running where you need to try very hard and can't tolerate 60 seconds of failure, you would bake multiple paths to a backend in the client configuration at some point. So the client would know if backend A is down it can do client side failover to backend B.
Sometimes this technique is used in conjunction with a highly fault tolerant server stuff, and not as a replacement for it.
And there is some beauty in it. It's fairly simple and with more servers in more places you get fewer affecting clients when one of the places fails and fewer disrespectful clients as well. It scales very well and cheaply.