Hacker News Reader

Question who's never relied on this: how do you test whether your multi-AZ setup can withstand an AZ dropping? It sounds like your third party apps make it kind of pointless to test anyway, but I'm curious!

View on HN

Ops type here, generally you have to fail an AZ and see what happens. If you have Dev/Staging environment that is exactly like production with plenty of tests, you can use that. I've seen only one environment actually like that though.

Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.

Thans! So do you have a staging environment for everything, then cut off the network or kill the instances, Aurora DB, etc.? Simulate some load while its going on, and then check for data loss? Then if that works try it in prod?

It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.

Yes, which means it's expensive which means companies don't usually pay for it so we should name and shame companies who aren't willing to pay that expense just to pad their bottom line.

I believe the article in question partially provides such a list.

Reply on news.ycombinator.com