Hacker News Reader

Comment by KeitIG | Hacker News Reader

I imagine Mr Good Guy at OVH telling some others:

"guys we have a single point of failure in our architecture with SBG, maybe we should...

- naaah it's fine, we do not have time nor resources"

Then shit happens.

edit: I have no idea what is happening exactly, but OVH being what it is, it seems extremely weird that all datacenters "can" get down at the same time, and it looks like a serious architecture problem to me (or backup systems, like generators, not being correctly tested... whatever). I am really curious about the future explanation with what happened exactly

edit2: Why all the downvotes? Even the status page of OVH is down, do not tell me it is good design. We are not here to be charitable, but realist.

View on HN

its OVH: The Hardware is good DDoS protection is good The Prices are high

but

support/administration does not work well, i have a lot of really weird story's with them, from them plugging in a keyboard in our server to reboot it (without any reason) to taking down a server for a requested maintenance only to notice after 4 hours of downtime that they did not ask their bosses if they were allowed to even perform the maintenance requested (and then not getting permission to do so after another 2 hours ..)

For me it feels like there are some really deep issues somewhere in the whole administration that make incidents like this no real surprise

Problem is most other providers dont work any better, so ...

Everyone makes mistakes, let's just hope they learn from it.

>ts OVH: The Hardware is good DDoS protection is good The Prices are high

The prices are high? Compared to what? Cheap is their raison d'être.

compared to other dedicated Server Hardware, not talking about business Cloud Infrastructure, no idea about that.

Sry if that caused confusion

Who's cheaper on the dedicated side? Hetzner can be a bit cheaper than soyoustart but wouldn't be aware of anyone else (with reasonable quality).

I've tried Hetzner but the Network Peering to Telekom is subpar compared to OVH. On Hetz I got about 40Mbps up/down to my local computer while on OVH I can easily load my DSL 100% without issues.

OVH is one of the few with direct peering to Telekom, most don't want to pay for that. Otherwise Hetzner is good, their own network is limited but peering works reasonably well. But apart from them I'm not aware of anyone with cheaper dedicated prices than soyoustart

I was considering their entire line...soyoustart, kimsufi, and ovh. There's not much cheaper than them. Hetzer in some cases, but not all.

Well, Hetzner for example.

Their prices are pretty amazing for some of the configurations you can put together, especially since you can pay a large setup fee in exchange for a lower monthly rate.

Just the bandwidth alone would cost us 3x what we pay at OVH if we were with one of the big cloud providers.

> - naaah it's fine, we do not have time nor resources"

Yup, been there multiple times in smaller hosting companies.

It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.

Calling OVH a "smaller hosting company" (which you did do indirectly) is rather funny.

Unless they meant "smaller hosting companies [than OVH]"?

The person you're replying to was relating their experiences at hosting companies that are smaller than OVH. How does that imply that OVH is a smaller hosting company?

to me it's ambiguous; it could be either of

> in smaller hosting companies [like ovh]

> in smaller hosting companies [than ovh]

Yes, a comma would have made it slightly clear:

> Yup, been there multiple times, in smaller hosting companies.

I find it funny that anyone would think I would or could refer to them as small (and get away with it)

This happens all the time. Every single thing that you see in software development happens in network engineering and data center engineering, except that where in development in general senior people who write software are capable of at least guestimating complexities to provide a marginally unified front against the unreasonable expectations of execs, it is pretty much never the case in neteng or dcops as those that develop software cannot write their heads around the complexities of working with physical hardware.

It is rather counter-intuitive. In neteng and dcops, "I don't know and I cannot find out. I can only attempt to mitigate what i think might have caused it for next time" is a very reasonable answer to 99% of the "why this happened?" questions because in order to replicate the situation to test the theory one needs to recreate the same problem again on the same scale.

This also means that certain things cannot be tested. Most of generator tests are garbage - turning on generator and running it without production load delivered over the transfer switch does not test anything other than that one can turn on a generator and run it. The problem typically happens not because the generator ( also is there the generator or the first and the second generator? Why is there no generator bank for a non monkey-sized company? ) does not start - the problem is because over time transfer switch develops a problem and unlike generators it is not possible to test a transfer switch where in the event of a test failure the customers won't lose power unless the data center is designed from the beginning to deliver A and B powers over separate circuits to every single customer and every single customer has per system ( not per rack ) transfer switches.

Of course it costs a lot more money, something that companies are reluctant to spend.

Any decent sr. network engineer or architect should be able to design you a network and explain the pros and cons, risks, and future scalability.

If someone doesn't know why a failure occured and they can't find out then they aren't looking hard enough

Please tell me about about this fabulous senior network engineer and where I can obtain a dozen of them.

I think I'm a pretty decent senior network engineer but I've been hit by firmware bugs more than once. No matter how much redundancy you build in, there's always shit that can go wrong and there will always be things that you just can't foresee (like the bug in this case). The software on these network devices and optical gear is written by humans, like all software, and is not perfect, like all software.

(In one case I experienced, the vendor reassured me that what I reported was not even technically possible -- until one of their engineers flew out and witnessed it firsthand.)

That seems a little uncharitable.

If the SBG issue really triggered the outage for the whole network I find it hard to believe that no one saw that problem beforehand. They probably thought that this was too unlikely to happen or that there are other failovers but never tested them properly.

No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. That can happen with one product (eg S3 failure) but datacentre switches should work even if the rest is on fire.

What is SBG? A CDN of some sort?

It's the location of one of their data centers. SBG for Strasbourg.

It's a collection of datacenters in Strasbourg.

Ah, gotcha.

So, that one datacenter caused all other datacenters to die..?

It's supposed to be two separate incidents: power going down in Strasbourg, and fiber network equipment going down in Roubaix (the main center of OVH's network) due to a "software bug".

It's explained here https://twitter.com/olesovhcom/status/928587258583748609 in French, they might post an English-language translation soon.

Thanks!

In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.

So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect.

They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.

The generators did work but they failed. Both of them.

I can not imagine they weren't tested.

But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.

When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"

Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.

I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents.

So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.

It doesn't matter how much testing you do – unfortunately some things can still fail.

I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.

Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.

What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.

To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.

If your generators are not reliable in the cold, the issue is the placement of them. This is something basic to account for.

Yes and no. What the generator vendor says when they sell you the generator and what actually happens 5 years down the line are two different things.

The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes.

While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature

It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that.

[1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...

Improbable. Even in a mostly-normal office building, the monthly generator test consisted essentially of "cut the mains power and see that no impact is perceived as UPS, batteries, and autostarted generators bear the load, in their turn for ~2 hrs total."

Europe sees that cycle every year, and have been for more than a century, so if they cannot handle that, they should be inside in a heated room.

Right, but that was obviously just an example.

Their generators are almost certainly tested frequently. But there could be any number of causes underlying the failure, and unfortunately sometimes failure does happen.

Generators in critical situations like this are usually provided with crankcase heaters which keeps the oil at temperature and makes them easy to start. Also, any emergency power system should be (and usually is) tested monthly if only to prevent the fuel lines going bad.

Data centers can go down while testing the generators too.

There was a DC in California a few years back that had three generators fail during a scheduled test and one of their server rooms had a blackout because of it.

This is a very good point. In all the companies I've worked in we've had more UPS outages than line outages as well.

I wonder if the generators or switching hardware had some kind of stupid IoT thing in them that required a network connection? That'd be one for:

https://twitter.com/internetofshit

> No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once.

It happened to GCE last year[1], though it only lasted 18 minutes.

[1]: https://status.cloud.google.com/incident/compute/16007?post-...

But that's one product, not the whole DC. As I understand the post, other services worked correctly during that time.

It was a power outage. Their own generators did not work. This explain why all is down, but there is surely something bad in their architecture.

A power outage in all locations? They claim to have "22 datacenters on 4 continents"…

EDIT: The title on HN in misleading, summary from their CEO here – https://twitter.com/olesovhcom/status/928592231807713280

Reply on news.ycombinator.com