Hacker News Reader

Comment by matthewmacleod | Hacker News Reader

That seems a little uncharitable.

If the SBG issue really triggered the outage for the whole network I find it hard to believe that no one saw that problem beforehand. They probably thought that this was too unlikely to happen or that there are other failovers but never tested them properly.

No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. That can happen with one product (eg S3 failure) but datacentre switches should work even if the rest is on fire.

> No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once.

It happened to GCE last year[1], though it only lasted 18 minutes.

[1]: https://status.cloud.google.com/incident/compute/16007?post-...

But that's one product, not the whole DC. As I understand the post, other services worked correctly during that time.

In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.

So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect.

They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.

When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"

I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents.

So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.

Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.

It doesn't matter how much testing you do – unfortunately some things can still fail.

I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.

Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.

The generators did work but they failed. Both of them.

I can not imagine they weren't tested.

But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.

What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.

To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.

Generators in critical situations like this are usually provided with crankcase heaters which keeps the oil at temperature and makes them easy to start. Also, any emergency power system should be (and usually is) tested monthly if only to prevent the fuel lines going bad.

Data centers can go down while testing the generators too.

There was a DC in California a few years back that had three generators fail during a scheduled test and one of their server rooms had a blackout because of it.

This is a very good point. In all the companies I've worked in we've had more UPS outages than line outages as well.

If your generators are not reliable in the cold, the issue is the placement of them. This is something basic to account for.

Right, but that was obviously just an example.

Their generators are almost certainly tested frequently. But there could be any number of causes underlying the failure, and unfortunately sometimes failure does happen.

The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes.

While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature

It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that.

[1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...

Improbable. Even in a mostly-normal office building, the monthly generator test consisted essentially of "cut the mains power and see that no impact is perceived as UPS, batteries, and autostarted generators bear the load, in their turn for ~2 hrs total."

Europe sees that cycle every year, and have been for more than a century, so if they cannot handle that, they should be inside in a heated room.

Yes and no. What the generator vendor says when they sell you the generator and what actually happens 5 years down the line are two different things.

I wonder if the generators or switching hardware had some kind of stupid IoT thing in them that required a network connection? That'd be one for:

https://twitter.com/internetofshit

What is SBG? A CDN of some sort?

It's the location of one of their data centers. SBG for Strasbourg.

It's a collection of datacenters in Strasbourg.

Ah, gotcha.

So, that one datacenter caused all other datacenters to die..?

It's supposed to be two separate incidents: power going down in Strasbourg, and fiber network equipment going down in Roubaix (the main center of OVH's network) due to a "software bug".

It's explained here https://twitter.com/olesovhcom/status/928587258583748609 in French, they might post an English-language translation soon.

Thanks!

Reply on news.ycombinator.com