Hacker News Reader

Comment by RamblingCTO | Hacker News Reader

I've been working on that for a client since yesterday (as a fractional CTO). Pretty hectic, basically nothing really works and we don't know yet if all data is lost or if anything is recoverable or when AWS UAE will become functional again so we can recover that region.

Finally, I have a very good argument for multi-region deployments ;))

that's my go to website atm: https://health.aws.amazon.com/health/status

View on HN

We didn't do multi-region deployments, but we did store database backups in a separate region just in case something really bad happened and our AWS region became unavailable. Also had a plan/some ready Terraform stuff in order to start setting up a deployment if it became apparent that the region wasn't coming back anytime soon.

IMO, if you're using AWS and not replicating your data somewhere else, this should be an eye-opener for you.

Not sure why everyone read this as me doing anything here, I'm a fractional CTO, which is kind of an advisor. Nothing invaluable will be lost tho. It's not the core platform, just a localized version for specific customers in the region.

Severity: Disrupted

So if data won't be recoverable you all will mark it something like "Status: FUBAR" or some equivalent term?

I wonder if ‘Apocalypse’ or ‘Molten Slag’ would be considered professional enough.

I'd accept "X_X" as status.

Status: Torpedoed

FUBAR, I would vote for that

What do you mean 'finally' - surely 'redundancy' or 'natural disaster' is reason enough.

To a lot of managers/startup execs this is something like "we won't ever need this". And I'd agree to some extent for not so important/rebuildable services. Depends on what you need. In startups, you don't have infinite time you can spend on stuff. But this makes a good case: if a geographical region only has one AWS region, don't keep data or run services that can't be easily rebuild somewhere else. In europe you can just pick two AWS regions and you stay in the same regions, UAE not so much.

There’s a reason literally every enterprise of any size in the early 00s not only had a DR site, but a DR plan they would test multiple times per year.

1. Just having a copy in two places doesn’t mean the copies are both good or that you can get DR online in any reasonable amount of time.

2. You quickly figure out all the things you thought “weren’t important” that prevent you from actually doing a successful DR test.

They amount of things that “cloud first” people just assume naturally takes care of itself because it’s in the cloud always amuses me.

You should have that for your main data, yes. I insist on that as well. Making backups, cold storage, and getting them back online. Yearly is not enough. My point was that not everything needs a backup or a DR. Spend your time wisely, we're talking startups, not enterprise. Can't talk more specifics obviously

The fact half the internet was down the last time US east was down tells me it’s not just startups that foolishly think they don’t need actively tested dr copies and plans when they move to the cloud.

I don't think this changes anything. It is always the same argument for me.

"How often do those happen?"

That was my point: now we have a very specific case we can argue. I always used the "what if a plane crashes into the data center" argument. Now I can say: in one of my engagements, we lost a datacenter completely because of a drone attack. That's a first for AWS as well, but we can draw from reality, not hypotheticals.

So AWS's availability zone architecture hasn't helped in this case?

If a whole region is bombed then you wouldn’t expect that region to stay up.

All AZs were affected

You don't already have a DR plan in place?

Of course, no worries. Nothing irreplaceable will be lost ;) It was meant as a general example to be used in future arguments for everybody in tech leadership.

Any backups?

> Any backups?

Yup something has to be said too about good old offline backups on fat tapes.

Reply on news.ycombinator.com