Ask HN: Is there a way to protect your app if heroku goes down? | Hacker News Reader

Ask HN: Is there a way to protect your app if heroku goes down?

1 pointbuthow3 comments

How do people build in redundancy/failsafe to an app being served on Heroku in case of an outage. I've had a good look around and I can't see a simple solution.

The main problem I'm coming across is the app I'm working on is built upon another platform that is using webhooks to communicate to the app. So all of the requests from the platform need to go to a static address (which means I can't redirect traffic to a 'cloned' instance of the app directly).

I've thought of building a switching proxy, having the platform point at the proxy address and have a task 'ping' heroku to see if it is up, and if not, change the endpoint of the proxy to send the requests to a cloned backup server. But building my own switching proxy seems risky in itself!

Does anyone know of a solution for this? What do you do?

Any help is greatly appreciated!

View on HN

How many levels of turtles do you want?

Multiple DNS entries for the hostname (i.e. multiple A/AAAA records) - most clients should re-try if one fails to load at all, but its not guaranteed behaviour, particularly if its not a browser.

Each A/AAAA record points to a virtual IP that is used by e.g. keepalived as a 'floating' IP between 2+ servers in that specific DC. Each A/AAAA record should point to a set of servers in a different DC, and potentially different provider/host too.

Each server that has this floating IP is a HAProxy node, which forwards requests to the various backend instances - normally this would balance the load, in case of issues it would become failover.

Thanks for you answer, you have opened my eyes! I didn't think to use the domain records. 2 levels would be enough considering I didn't know you could do more than 1 level.

Do you know a good starting resource to set up any of those methods?

So keep in mind, the multiple "levels" are about having redundancy.

So if you make something singular instead of multiple, it becomes a single point of failure.

The dns part is easy but the most variable as it relies on client behaviour.

haproxy/similar to balance between eg two servers is simple enough, but if you want haproxy itself fault tolerant you need stuff like vrrp using keepalived or similar. Thats less easy and is dependent on your hosting provider.

Of course you can also find "hosted for you" resources too. Many bigger "modern" hosting companies (ie rented virtual servers and hosted services rather than shared hosting) have some kind of "load balancer" sevice. Eg aws/google/azure will all have an offering of some kind, as do the likes of Linode, Digital Ocean etc.

The last two also offer "floating ip" (essential for vrrp - running your own load haproxy cluster)

I havent checked explicitly but I'd imagine at least some of those hosted loadbalacers only work with their local networks.

I hope this is clear, happy to discuss it further directly if you wish.

Reply on news.ycombinator.com