Hacker News Reader

Comment by dx034 | Hacker News Reader

So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect.

They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.

View on HN

When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"

I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents.

So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.

Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.

It doesn't matter how much testing you do – unfortunately some things can still fail.

I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.

Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.

The generators did work but they failed. Both of them.

I can not imagine they weren't tested.

But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.

Reply on news.ycombinator.com