This seems to significantly under-report what's going on, see:
https://www.theregister.com/2023/04/26/google_cloud_outage/
There is mention of a fire as well.
https://www.theregister.com/2023/04/26/google_cloud_outage/
There is mention of a fire as well.
A few years ago I implemented a top to bottom ISO27k1 ISMS for a client handling extremely sensitive and mission-critical data for industry.
One risk I recommended controls for was that of a fire and/or flood at their primary datacentre for their client-facing offerings - this datacentre. I’ve experienced the misery of a datacentre oops myself, firsthand, twice, and it’s a genuine risk that has to be mitigated.
At my insistence, I had them burn hundreds of man-hours ensuring that they could failover to a new environment in a different datacentre with a bare minimum of fuss, as what I arrived to was an all the eggs in one basket situation. It took a fair bit of re-engineering of how deployments worked, how data was replicated, how the environment was configured - but they got there, and the ISMS was put into operation, and was audited cleanly by a reputable auditor, and everyone lived happily ever after.
Except… they were acquired by private equity. Who had no truck with all of this costly prancing about with consultants and systems. Risk register? Why do we need this? What value does it add today? ISO27k1? Don’t be silly. We have that certificate. You don’t need it. Dev team, ops team, leadership — almost everyone — ejected and replaced with a few support staff.
I see their sites are down.
I am sure that my name is currently being damned in a boardroom in Chicago, as as the person who warned of this, I am likely seen as responsible.
In that sense nothing ever changes. Anyway, kudos to you for staying the course on topics like these.
The fact is, most cloud providers offer multiple regions, which have the capability of giving you more geographic redundancy than most companies that operate in their own datacenters have.
Whether you choose to adopt a multi-region or multi-datacenter architecture is really orthogonal to whether you choose cloud or on-prem.
Also, why batteries in a datacenter? When you implement a flush() command at the lowest level you're faced with two choices: 1) actually write to disk, then return from the call, 2) write to some cache/RAM and have just enough battery locally to ensure that you can write it to disk even if all power goes out.
Then there's the other problem of surviving long enough between a power interruption and diesel generators starting up. But this is a smaller problem, rebooting all instances in a datacenter is less bad than losing some data that was correctly flush()ed by software. Bad flush() behaviour can result in errors that cannot be recovered from without a complicated manual intervention (for example if it causes corrupted and unreadable database files).
Everything serious in the telecom/ISP infrastructure sector has a big -48VDC battery plant, or preferably separate A and B side -48VDC battery plants, to provide a significant buffer between power going Grid --> AC-to-DC Rectifiers --> Equipment, and when a generator can start up, warm up, and transfer switch does its job.
Even if a bunch of servers don't have any UPS or battery backup because they're designed to tolerate individual node (or whole rack, or whole row failures) the core network equipment in a datacenter will still have a huge battery plant.
Ideally if you have a chilled water loop for cooling you do not want it anywhere near your big-ass racks of batteries. Or near the racks that contain the rectifiers and DC breakers, distribution bus bars.
If you look at the battery racks in a traditional telco CO in the US for instance you will see that all of the cabling and batteries are a minimum of 1 foot off the floor, so that the whole place could theoretically flood and the DC distribution would remain unaffected. Same principle that applies to very traditional setups with wet-cell 2V lead acid batteries also applies to more modern things if building from scratch.
Think like a fiber termination demarc vs an in-cluster mesh.
By square footage maybe 95% of a google DC might have no UPS or battery backup but the core network for things like routers and DWDM transport equipment absolutely will have such.
If they were unlucky enough that the burst cooling loop met with the battery plant for the core gear in a building or small campus of buildings....
They create 3 separate arrays of batteries in each back. Each array represents a power phase, A-B-C.. if I remember correctly, each array has a number of low voltage/2000 amp batteries connected in series to make up for a 2000amp 480 volt leg on the other end.
In a tier 4 plus+1 datacenter, they have 4 battery rooms and 4 generators for each data pod. You have a primary generator and UPS battery set, and a backup generator set for each pod. And then that generator set has its own primary and secondary backup set. The end result is that they can work on any piece of equipment without interrupting power. In the event they lost the primary set or needed to take it offline for maintenance, they have the whole secondary redundant set to fallback on.
The servers on the received on the power cord after it passes the switchgear never know that there has been power source changes on the other end.
I think this is sort of big.