Hacker News Reader

Comment by ovi256 | Hacker News Reader

The report points out (Image 8) design choices that contributed to a raging fire, once the fire started:

* no emergency electricity cut-off device, an "economic strategy choice of the site operator" - the electrical room where the fire started was hot, 400C at the door (measured by thermal camera), with meter long electrical arcing from the door, thundering deafening sounds. Making access of utility technicians "difficult" - it took them 2 hours to cut incoming electrical service from the utility. On site UPS devices also had no cut-off, so they kept supplying.

* emergency water network provided only 70m3/h at the site. A firefighting boat arrived, Europa 1, was called, supplying 14.5m3/min max flow rate.

The freeflowing air cooling design, a good design choice as it saves on operating costs for cooling, contributed to nourishing the fire.

View on HN

So basically, my takeaway is:

* Freeflowing air in DCs is good for cooling( * It’s bad when you have a fire;

* An improved DC design would allow an operator to $somehow stop the freeflowing air (although one could argue that it’s not free flowing anymore if one can control it);

* I’d like to know how much money really was saved by not allowing the UPSes to be cut off.

I’m very curious how the insurance companies respond, and whether they’ll demand e.g. UPS supplies to be able to cut off. Or maybe in general, the fire department should be more aware of these types of trade-offs being made, and give their approval accordingly.

It is VERY common in buildings to have ventilation dampers that are closed when a fire alarm starts exactly for this reason. I am most surprised that the design sounds like it was made a certain way deliberately even though the basics were got wrong. Does Strasbourg not have any fire regulations that require fireman accessible power cut-outs, automatic extinguishers etc?

It doesn't matter just that their equipment burned but the severity of the fire was massively dangerous to the fire brigade and anything else nearby if it couldn't be extinguished properly.

> Does Strasbourg not have any fire regulations that require fireman accessible power cut-outs, automatic extinguishers etc?

I think it's a safe bet it doesn't, because that building passed fire safety review and was approved as such.

I think it’s a valid question; inspections themselves can be failure points.

I can't speak for Strasbourg, but here in the UK you don't get an inspection by the local building inspector or fire department - instead, you hire a fire risk assessor of your choice, on the free market.

So if you're not in compliance, you don't need to bribe anyone or cheat - you simply have to hire a friendly dumbass who'll check the basics but won't ask too many questions.

Mark this one up as something else that's been privatized in the UK that's shocking to me as an American. This sort of stuff is always run through the government (the level depends on the state).

Insurance companies might impose additional requirements, but then they'll send out their own inspectors.

The inspector is a chartered engineer, and is liable if your building don't confirm to the fire code.

>Instead, you hire a fire risk assessor of your choice, on the free market.

That sounds a lot better than the one size fits all crap we get in the US. If you are trying to do anything other than take over an existing and operating industrial site that's grandfathered in or run a warehouse for some sort of boring and non-reactive goods it tends to be a terrible maze of conflicting and irrelevant requirements.

Note the OP said the inspectors were privatized, not the actual building codes. The only difference is the private inspectors sound like they'll let you skate by on some stuff.

You find the same dynamic in states with private emissions inspections. Spend any time on a car enthusiast website and everyone knows "the guy" to go to that'll pass your obviously illegal emissions equipment bypass...

>Note the OP said the inspectors were privatized

I know exactly what he said and that's why I said exactly what I said.

>The only difference is the private inspectors sound like they'll let you skate by on some stuff

Government building inspections are incredibly onerous unless you're a homeowner building a deck or a developer building cookie cutter houses or offices. You run into all sorts of stupid edge cases depending on what the facility being built is intended for and you need to get clarification on what you should do. Have you ever actually tried to get government clarification about something? They need to be dragged kicking and screaming, often through a courtroom to ever clarify anything in writing because nobody wants to set a precedent or take the responsibility of making judgement calls. Having professionals who will take the responsibility of those judgement calls is a massive net plus.

The fact that your knee jerk response is to frame professionals using judgement their jobs as "letting you state by" is just crazy. Would you say the same thing if the context was a PE crunching numbers, finding something that didn't pass the default rules and then using their judgement to conclude it was fine because of other situational details? People do their jobs satisfactorily the overwhelming majority of the time. Being on the government payroll doesn't make public inspectors immune from playing favorites or making screw ups. It just means the oversights follow a different pattern. Everyone's experienced plenty of government inspectors who don't do their job because it's Friday and they want to GTFO.

> Spend any time on a car enthusiast website and everyone knows "the guy" to go to that'll pass your obviously illegal emissions equipment bypass...

If the vehicle code were written the way the building code and occupational health and safety code is that's the only way any commercial vehicle that isn't a cookie cutter box truck would ever pass an inspection.

> The fact that your knee jerk response is to frame professionals using judgement their jobs as "letting you state by" is just crazy.

The UK system (builders choose their own building control inspector in the free market, building owners choose their own fire risk assessor in the free market, makers of building materials choose their own flammability testing house in the free market) produced hundreds of tower blocks clad in flammable cladding and insulation, and a fire in which 72 people died.

After all, why choose the guy who's going to give you a hard time, when you could choose someone friendly, helpful and trusting?

And inspectors and testers know that. And those that are still in business are so very friendly, helpful and trusting, in fact, that the person certifying whether your insulation is flammable will let you set up the test rig yourself, and they "won't notice" you've put magnesium oxide fireproof boards over the flame sensors.

> Does Strasbourg not have any fire regulations that require fireman accessible power cut-outs, automatic extinguishers etc?

Fire codes for industrial buildings, office buildings and residential buildings differ by quite a lot. Usually, they get stricter the more people are supposed to be in a building and what their training status in firefighting/escape is.

> Does Strasbourg not have any fire regulations that require fireman accessible power cut-outs, automatic extinguishers etc?

On the other hand, it could be that this was deliberately allowed for buildings with low/no occupancy, easy, safe & quick escape routes and no risk to neighbouring properties.

If someone wants to risk their own property, as long as they don't risk harming anyone, why not allow it?

Just don't try to buy fire insurance.

>If someone wants to risk their own property, as long as they don't risk harming anyone, why not allow it?

Because it's a flawed premise, you don't exist in isolation from the outside world.

The smoke is harmful to everyone nearby, especially a data center fire where you have burning servers, UPS batteries, etc. contributing large quantities of burning cadmium, lead, mercury, assorted plastics and so on. The environmental impact from the fire event alone should make this a non-starter.

Also, little to nothing can be recycled after such an event, and replacing the lost equipment requires a whole bunch of environmentally damaging mining and manufacturing processes.

>no risk to neighbouring properties.

Even if you plan carefully there is an unacceptable risk of fire spreading. And while some datacenters are moving to the countryside or aquatic environments, most are in built up areas for easy access to industrial power supply, workers, spare parts, freight, etc... And I have no confidence that a company who's disaster plan is "screw it, let it all burn" would adequately maintain firebreaks and other containment measures.

Frankly, I find the very idea of such excessive and deliberately considered wastefulness to be grotesque.

> I’d like to know how much money really was saved by not allowing the UPSes to be cut off.

I don't think this was a cost-cutting measure. Instead, (automatic) cut-off devices are also systems that can fail. And if there is such a centralized device, it can nullify the hard work of 4 redundant power supplies if it "decides" to malfunction one day. So, this was a SPOF-cutting measure. I still think that having 4 separate switches for 4 power supplies is not so wrong. Firefighters anyway wrote that the electrical room had a door at 400C and observed 1-meter arcs, so they had to ask the electrical company to cut the power from their side for the sake of safety anyway.

I disagree with your idea that not having an automated power disconnect system is acceptable but for the sake of argument lets say it is. The standard for manual emergency power cutoff switches is to be located where they are easily accessible in an emergency which means outside of the electrical room. Most large data centers have manual switches in multiple locations. They should at a minimum have had cutoff switches located in their NOC.

Right! Sorry, I didn't write it clearly but I agree that there should be automatic disconnect, but think it's OK to have 4 separate disconnects to maintain the supply redundancy. Though now it can get really expensive.

Building datacenters and is hard. So this is an opportunity for cloud providers.

1. Imagine if Amazon starts building datacenters for others as a service

2. People starts leasing or renting entire DCs built by Amazon, Google etc. Maintenance is done by these cloud providers.

Do we have these already?

> 1. Imagine if Amazon starts building datacenters for others as a service

Just as an FYI, those fancy datacentres you see on Amazon/Google/Microsoft marketing videos are really the exception rather than the rule.

You will find the majority of cloud servers in exactly the same third-party datacentres that everyone else uses.

Why ?

Because only the US and perhaps one or two other countries in the world has the spare land for a cloud operator to dump a massive datacentre campus on.

Most other countries don't. Or if they do, its either protected land (greenbelt etc.) or its uninhabitable (e.g. Australian Outback).

The observation in the first half of your comment seems sound but your explanation doesn't make sense.

There's rather a lot of land in the world outside of the US that could have a datacenter built on it. There's also a lot of Australia that's uninhabited but not uninhabitable.

I oversimplified a bit, by "land" I also meant associated infrastructure, which includes for example access to electricity. In most countries, building out new high-voltage infrastructure to where it does not exist is both financially expensive and technically painful (planning permission etc). Same for pulling fibre runs to the middle of nowhere.

No doubt other things like local laws, tax rebates and whatnot also come into play as well.

You may seek to argue that there are a small number of third-party datacentre sites where the cloud operator is the sole tenant of the building. But this again is not the same thing as the cloud operator building their own. They get the option to up sticks and leave at the end of their contract. They also don't have any responsibility over facilities management etc.

At most sites, the cloud operator simply has whole or part of a floor (or floors in larger buildings), the rest of the building is occupied by third-party customers.

Your order of operations is inverted. The existing data center providers have been doing this for a few decades. Equinox, Dell, and friends are the ones who do the build out, management, etc. If youre big enough theyll also build to suite, though at that size you probably have a lot of in house expertise as well.

AWS, Azure, Google Cloud, etc are the alternative to dealing with dedicated physical infrastructure and financing. Things like AWS Outpost are there to bridge the gap for companies/workloads that dont fit existing public cloud offerings.

The report says that the cut-off device was not present due to an economic/financial decision made by the company . It's weird to see that in a post incident report from a fire department, but I guess they thought it was worth mentioning.

> The freeflowing air cooling design, a good design choice as it saves on operating costs for cooling, contributed to nourishing the fire.

The reason you use forced air in your house instead of being built for natural convection is so you don't die in a fire. Fires are all about convection. Infernos doubly so.

There was an early luxury cruise ship tragedy, I think in New York City. A 'freeflowing air cooling design' and all wood paneling. It caught on fire, so they turned around to come back into port... and burned to the waterline, killing a bunch of people.

Building code for passenger ships got changed to require forced air and limit natural (flammable) materials after this.

Reply on news.ycombinator.com