Hacker News Reader

Comment by a2tech | Hacker News Reader

Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm and their systems are not designed for speedy restoration into another region.

View on HN

Data Center problems are Azure's issue, not the VSTS team, so your criticism should be focused there. Even though Microsoft runs all of these services that's out of scope for a VSTS blog.

They linked to the Azure service details in the Azure Status link if you wanted more information: https://azure.microsoft.com/en-us/status/history/. It sounds as though they're compiling a report on this problem which will be released in the coming weeks.

> Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm

This shouldn't matter, only to them and whoever operates the datacenter.

> and their systems are not designed for speedy restoration into another region.

This is what matters. You have a cloud service that is not really designed for the cloud.

Seems like there's more information in the preliminary RCA on https://azure.microsoft.com/en-us/status/history/

> Initially, the datacenter was able to maintain its operational temperatures through a load dependent thermal buffer that was designed within the cooling system. However, once this thermal buffer was depleted the datacenter temperature exceeded safe operational thresholds, and an automated shutdown of devices was initiated

Total failure of a data center's cooling apparatus seems to be a very rare occurrence to me, perhaps limited to simultaneous failure of utility and genset power (example: electrical switchgear and fuel pumps underwater due to flooding).

Anyone have any data around how frequently such a failure occurs?

I had the same thought. The only thing I could come up with is that it wasn't a failure of power supply, but that a surge took down enough cooling systems that they couldn't maintain temperature. A lot of DC's I've seen are N+1 with cooling (or even 2N), but they all run at the same time and are the same units. Or the control system went down, and they weren't able to get it back up and running, although I would think they would have redundancy in that case.

Reply on news.ycombinator.com