Comment by JagMicker | Hacker News Reader

Comment by JagMicker | Hacker News Reader

Maybe folks will start to realize that an EDG (Emergency Diesel Generator) without critical support systems is useless.

I was just reading about the Fukushima accident, and how most of the EDG's failed because they were water-cooled, and while the EDG's were located out of harms way, the pumps for providing cooling water were located on low ground and were damaged by the tsunami. http://fukushima.ans.org/report/accident-analysis

Yesterday I saw a story about the Datagram data center in NYC having to shut down. From their reports about the damage (http://www.datagram.com):

"As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency systems at 33 Whitehall, NYC fully staffed and awaiting the storm to hit Manhattan's shores. Once ConEd lost power to Lower Manhattan, Datagram's emergency systems kicked on maintaining power to Datagram's datacenter. Unfortunately, within a couple hours of the storm hitting Manhattan's shores, the building's entire basement, which houses the building's fuel tank pumps and sump pumps, was completely filled with water and a few feet into the lobby. Due to electrical systems being underwater the building was forced to shut down to avoid fire and permanent damage."

It's pretty obvious that, despite all the disaster planning done in the past, Datagram (and TEPCO, and others) have really neglected to appreciate the potential modes of failure for their backup power systems. In both cases, they misunderstood the threat to critical backup power infrastructure. If your EDG is on the roof but the fuel pumps and electrical switchgear is in the basement, what will happen during a flood? If your EDG's are on high ground but the pumps to cool them are not, what happens during a tsunami?

Unfortunately, physics dictates this setup. To get to a top floor generator (which is generally necessary for ventilation), you have to pump from the bottom. For most buildings, that means from the basement, where the fuel tanks are placed for other obvious reasons. (Consider if one started leaking, would you want it dripping down to all of the floors below it?)

These data centers are well designed, but it's impossible to cover every disaster scenario.

It's easy for you to sit here after the fact and snipe at them. If you really think you can do better, go start designing them yourself. If you succeed, you'll do quite well for yourself.

Regardless of whether they can be designed better, the data centres were assuring everyone that everything will be fine because they have backup generators, ignoring the probability that flooding would take out those generators.

If I had a website or service hosted at a data centre likely to go offline, I'd be annoyed if I were reassured that everything was taken care of right before the system went offline.

Your one complaint about diesel fuel dripping is not a big deal in the short term. Diesel fuel is quite safe and hard to ignite. THere are plenty of places/situations where significantly more toxic substances are stored in tanks that are not in basements. Your logic would apply more there (could you imagine if acetone was dripping from the ceiling?). As a practical matter, anything leaking through floors is really bad, but the problem is solvable and not a good reason to keep fuel tanks in a basement.

So what other reasons are fuel tanks placed in basements that outweigh the risk of putting them in the basement (everything except tornadoes would seem to be a con for basements)?

Additionally, "Flooding" would seem to be a fairly typical disaster scenario. Again, what does putting them in the basement prevent against that makes it a good alternative.

Plenty of people are designing better datacenters and using them, there is no need for the parent to go out and do anything. If customers hold internap/et al accountable, the problem should take care of itself.

(of course, they won't, but that's another matter :P)

It's easy to say diesel fuel is "quite safe and hard to ignite." But if the NYC Fire Code says you have to store it in the basement, then guess what? You have to store it in the basement

Then either

1. Stop claiming they are disaster tolerant, because they aren't. Flooding is a common case.

The exact rule i'm aware of is "406.5.3 Storage tanks. Motor fuel storage tanks shall be installed below ground, except as authorized by the rules of the Fire Department"

So get the fire department to authorize it.

2. Datacener folks should lobby new york city. Bloomberg would likely care.

3. Find something other than motor fuel, which is what is restricted :)

If all else fails:

4. Stop building disaster tolerant datacenters in new york. This is why a lot of financial folks who aren't HFT's build in Seacaucus, etc. There are very few people who need to place a DC in new york, and can't afford to be across the river.

(I'm very keenly aware of the bandwidth situation in both places, FWIW)

I'm sure if datacenters start moving across the river, Bloomberg would care.

Building codes and other restrictions are taken into account by the big boys (Google, Apple, Facebook, etc) when they build large datacenters.

Fuel pumps can be installed such that they are resistant to flooding. A water-tight seal on the fuel tank, sealed power line from the generator, and other straightforward water-proofing measure would have kept Peer 1's generator working as long as the building's fuel supply held out.

Manhattan is a low-elevation coastal island in a region of the country prone to hurricanes. Flooding is an obvious disaster scenario. It's not like we're asking them to plan for nuclear war, here.

There's no reason the fuel tanks and pumps can't be in the basement and still withstand a flood if they're waterproof, which would be relatively easy to achieve. I suppose these systems are typically designed to mitigate blackouts though rather than natural distasters, and floods were not considered as a possibility. As you say, it's easy to be critical after the fact.

How about coming to the reasonable conclusion that a data center in a 17 story tall building is not practical? Most purpose built data centers are only a few stories tall and don't have these type of issues as they are not built on islands close to sea level.

For those companies hosting production services on an island a few feet from sea level without a DR plan, I simply laugh. You deserve our scorn, not our production data.

As someone who designs these sorts of things for a living, there are a host of problems with having diesel fuel on the room. For one thing, fire codes limit the amount of diesel fuel you can store on the roof (for obvious reasons). For another thing, the trucks that carry in the diesel fuel to refill the tanks are located on the street. One way or another you're going to have to pump fuel from the street up to the roof, which is going to leave the pumps that do so susceptible to flooding.

Would a closed, pressurised air system work as a one-off "push all diesel from this tank to the roof" system?

So you could store:

- tank full of diesel

- some compressed air cylinders

at a low level, and use them in the "everything is submerged" event.

My expertise is on the electrical side, rather than the plumbing side, but I suspect the amount of compressed air you would need to propel 500 gallons or so of diesel up 17 stories would be impractical.

One can buy pneumatic submersible pumps. I could see having a compressor on the upper levels of the building drive the pump in the basement.

http://www.atlascopco.us/usus/products/navigationbyproduct/p...

Of course, if the diesel has water in it it's no good anyway. And the submersible pumps are mainly sludge or slurry pumps and (I suspect) not rated to handle fuel.

500 gal diesel = 1622 kg

17 stories = 46 meters

Energy required = 1621 * 46 * g = 732 kJ

Integrating PV=nRT, W = -nRT*ln(V₂/V₁)

assume you're going to compress the air to 150 psi, or 10 atm, so V₂/V₁ = 10. Assume T=10 degrees C. Solving for n, you need 880 moles of air.

Putting this back through PV=nRT, we get a modest 190 gallons of air (compressed).

So you need a 200 gallon tank with 150 psi storage. Look, here's one on craigslist for $400.

http://eugene.craigslist.org/gms/3306858286.html

Given a 200 gallon tank with 150 psi pressure you would no longer be able to pump the diesel to the roof after 460 gallons of the diesel is pumped, assuming temperature is constant and line friction losses are negligible.

Also, I think you have made an error in your arithmetic.

I would have thought Tropical Storm Allison in 2001 was the teaching moment for the data center business, when all the underground infrastructure of downtown Houston was completely flooded. Apparently planners had not realized that critical backup power systems (or research animals, or symphony archives, or ...) shouldn't be below the level of potential flooding. Houston is 25 miles from the Gulf, so this wasn't a storm surge, just a huge steady rain event. I guess the lesson will be repeated until building management learns.

http://en.wikipedia.org/wiki/Tropical_Storm_Allison

It's hard to call the diesel generators "useless" when this company is successfully using them to power its facilities. The bucket brigade is clearly not an optimal solution, but it is a solution that is working so far.

Do you have better suggestions? (Not being a troll - you make a fair point and I'm genuinely interested to hear what you have to say if you would have done it differently)

To reply to mjwalshe ... who from the looks of it is hellbanned:

  > Submesible pumps or rig a block and tackle out of a window and haul jerry cans up that way safer thna carying open buckets by hand.

Diesel fuel can only be ignited by a high power flame (such as from a propane torch), and will not simply ignite due to sparks. Diesel fuel is amazingly safe to handle, much like fryer grease is safe to handle.

True but a slipping on spilled diesel and falling down a flight of stairs is one danger still think rigging block an tackle would be much more efficient.

Shouldn't it be possible to make the fuel pump and it's power supply a closed, water proof system? Such that the fuel tank could be flooded and the pump would still run?

Probably a good idea, but I can't help but think what other weakness that system might turn out to have in the next disaster. I don't know if it's just me, but I was definitely very surprised by the degree of flooding - I would have expected debris, structural damage and power outages, but even as a former New Yorker, I was surprised it all happened.

No, I don't. But I think this shows a lack of understanding on the part of the data centers. Datagram said they'd tested their generators prior to the storm, but apparently they were unaware that the electrical switchgear was located in an area that would readily flood. During a flood, what good is a generator if the electrical equipment can't be used? Would a possible solution be to have the generator output run directly to the datacenter, and to have the ability to isolate it from the building ("mains") power?

Also, see my sibling response to Pyrodogg - it was kind of a joint response to both :)

Refraining from locating a data center in a floodplain seems like the easiest way to get greater reliability.

Reply on news.ycombinator.com