AWS Frankfurt incident
status.aws.amazon.com
status.aws.amazon.com
Starting at 1:18 PM PDT we experienced connectivity issues to some EC2 instances, increased API errors rates, and degraded performance for some EBS volumes within a single Availability Zone in the EU-CENTRAL-1 Region.
At 4:26 PM PDT, network connectivity was restored and the majority of affected instances and EBS volumes began to recover.
At 4:33 PM PDT, increased API error rates and latencies had also returned to normal levels. The issue has been resolved and the service is operating normally. The root cause of this issue was a failure of a control system which disabled multiple air handlers in the affected Availability Zone. These air handlers move cool air to the servers and equipment, and when they were disabled, ambient temperatures began to rise. Servers and networking equipment in the affected Availability Zone began to power-off when unsafe temperatures were reached. Unfortunately, because this issue impacted several redundant network switches, a larger number of EC2 instances in this single Availability Zone lost network connectivity.
While our operators would normally had been able to restore cooling before impact, a fire suppression system activated inside a section of the affected Availability Zone. When this system activates, the data center is evacuated and sealed, and a chemical is dispersed to remove oxygen from the air to extinguish any fire. In order to recover the impacted instances and network equipment, we needed to wait until the fire department was able to inspect the facility. After the fire department determined that there was no fire in the data center and it was safe to return, the building needed to be re-oxygenated before it was safe for engineers to enter the facility and restore the affected networking gear and servers. The fire suppression system that activated remains disabled. This system is designed to require smoke to activate and should not have discharged. This system will remain inactive until we are able to determine what triggered it improperly.
In the meantime, alternate fire suppression measures are being used to protect the data center. Once cooling was restored and the servers and network equipment was re-powered, affected instances recovered quickly. A very small number of remaining instances and volumes that were adversely affected by the increased ambient temperatures and loss of power remain unresolved.
We continue to work to recover those last affected instances and volumes, and have opened notifications for the remaining impacted customers via the Personal Health Dashboard. For immediate recovery of those resources, we recommend replacing any remaining affected instances or volumes if possible.
The paper records they destroyed by blowing up a whole building, with hacked UPS systems. [0]
[0] https://www.geekwire.com/2016/mr-robot-rewind-pwning-battery...
sounds like on a submarine. On submarine the people who couldn't make it out of the sealed section in time, would use the personal breathing device - i wonder if AWS has such thing in the datacenter, and how do they make sure that all people are out from a huge datacenter. Several years ago a Russian nuclear sub on a sea trials after refit had the fire suppression system activated in the section where a bunch of people gathered - crew and the navy yard civilians, probably they celebrated drinking and smoking - and insufficient number of the breathing devices combined with lack of skills to quickly use it on the part of the civilians resulted in 20 deaths and 40 injured (lung damage, etc.), mostly the civilians.
That, plus you typically dont go and hangout on the DC floor, only the personel who is really required gets to go in.
times changed. I remember my friend bunch of years ago complaining to me that the DC at their branch of a large well known transnational was the favorite place of the employees for various informal activities, including that he was constantly dealing with used drug paraphernalia being left all over the place.
Answer to the comment below: yep, mostly syringes/needles.
Holding doors was always a problem, but I guess that nowadays badges can have RFID or some other kind of wireless tracking so even if you hold the door, people in the control room can always know how many people are in each room, all the time. It's like 6 years now since the last time I was in a datacenter...
Door locks during an emergency are usually denoted to be fail-open so someone doesn't get trapped, though that may differ depending on the security level of the facility.
The Smoke Detector cloud instance was down for some reason so the system had to go with temperature!
Also they didn't have breathing gear (and trained staff) so you could go in and restart without waiting and also in case of an accident being able to try and rescue people.
Back when worked in RnD for the lab where we could have had a Freon leak we had breathing gear just outside and some people trained to use it
I'll revise that to - I hope that whomever is relying on AWS for safety-critical systems at least does it over many regions. It's still dumb, because even AWS occasionally has global/multi-region outages, but at least it hopefully reduces the chance for it.
Wtf, why not? It's drastically easier, and probably cheaper, to achieve that level of redundancy with AWS than doing it yourself.
> It's still dumb, because even AWS occasionally has global/multi-region outages
Really? Like when? The only potential one you can claim was multi-region, was when S3 us-east-1 was down, and with the old default behaviour - if you didn't specify where your S3 bucket is it would pass through us-east-1 to ask where it is - that impacted lazy code that had nothing to do with us-east-1. That's almost entirely on developers and such though, so hard to claim it was a multi-region or global outage.
At that point without actually going in and checking they have no real way of knowing if there really is/was a fire or not. So the proper procedure is to let the professionals handle it (wait for the fire department to clear the building). No amount of server downtime is worth sending a "not a firefighter" into a possibly burning building.
And facilities like these have strict control of where people can be so they know if someone is in there or not without of going in to check.
Nevertheless, having breathing gear could allow beginning recovery action just after the fire department finished procedures. This would shorten recovery time.
I would hope that they do audit people in and out so i case of accidents you can account for everyone.
Oh and these would be trained people
No noticeable smoke, but that does sound a bit closer to a fire than they planned for.
Either some system(s) not shutting down when the ambient temperature becomes high, or the ambient temperature was so hot it damaged powered-off equipment (!).
Thermal expansion/contraction is one thing - a solder joint that was marginal might have contracted enough to actually break.
I guess thats something hard to test for , let’s put this switch in an oven and run it to make sure it auto powers down at 80c :) every time we release a firmware.
was a problem from multiple vendors who issued bios / firmware patches. Cisco, dell, Emc, brocade.
These may be redundant by a reasonable definition, but they clearly weren't here....
We have restored network connectivity within the affected Availability Zone in the EU-CENTRAL-1 Region. The vast majority of affected EC2 instances have now fully recovered but we’re continuing to work through some EBS volumes that continue to experience degraded performance. The environmental conditions within the affected Availability Zone have now returned to normal levels. We will provide further details on the root cause in a subsequent posts, but can confirm that there was no fire within the facility.
https://www.merriam-webster.com/dictionary/fire-sale (": a sale of merchandise damaged in a fire")
that's not good for computers either
“The EU ban on the use of Halon in fire extinguishers actually came into force in October 2000 and was implemented in the UK in 2003, as a result of scientific research linking Halon and other CFC’s to Ozone depletion. The ban in practice is not total.
Existing owners and users of Halon 1211 portable fire extinguishers may be able to claim exemption to the EU ban for certain “Critical Uses”.
Broadly speaking this includes limited applications within the aircraft industry, military / armed forces, petrochemical industry and some specific marine applications“
Chances are this facility was built after 2000. I think that makes it extremely unlikely it can claim exemption.
See also https://en.wikipedia.org/wiki/Montreal_Protocol, which says
“The Vienna Convention and the Montreal Protocol have each been ratified by 196 nations and the European Union, making them the first universally ratified treaties in United Nations history”
I assume so but would be nice to know its working as expected!
I wonder whether its because a large load of capacity needs suddenly in unaffected AZs put abnormal stress on things like networking to the whole unaffected AZs.
My only experience there is in Azure, when they deployed patches for the Heartbleed etc issues, for a few weeks things were much slower in CPU power (our response times just shot up 20% for no reason then recovered a few weeks later) and there were network related timeouts that were abnormal and it all settled down eventually.
For reference https://heartbleed.com/
"To ensure that resources are distributed across the Availability Zones for a Region, we independently map Availability Zones to names for each account.".
my eu-central-1a is not necessarily yours.
[0] https://docs.aws.amazon.com/ram/latest/userguide/working-wit...
[1]: https://docs.aws.amazon.com/ram/latest/userguide/working-wit...
We also do have some EC2s running with nodejs applications and there the aws-sdk just errored out with "UnknownError: 503" and simply stopped logging until we restarted the machines. The machines itself were not stopped at all.
Other than that, I can't see any effects across our accounts. Also not RDS or anything else. Fascinating. Glad its under control and seemingly nobody died or so.
We have mostly found one or two classes of jobs in the orchestration for which nomad stopped retrying deployments before the ec2 instances running the allocations were fully failed and removed from the cluster - and our on-call was unsure how to handle that situation in nomad right. Network and routing were really weird at some point. Additionally, we ended up with a couple of container instances orphaned from the container management, which was strange for a moment.
This was made a bit more hectic over here because a second hoster apparently fried their own network at the same time so we needed some time to realize we have two issues.
Overall, 5/5 Outage, would fail again once we've updated our jobs. We're happily close to not caring about such an incident.
Amazon had released a version of their AWS Linux edition that rebooted randomly due to a kernel bug, and I was working on our staging cluster, but I didn't even notice that I had EC2 instances that randomly rebooted and dropped because Nomad just kept the workload up.
The air was too hot to breathe.
I had to take and hold breath to jump in to shut down most important assets.
There was redundant air conditioning except for exactly one element: air inlet on the roof. A thin plastic bag blocked it.
Nice case to prove that plastic waste is hazardous though!
But the offset was definitely increased understanding of the concept of single point of failure.
Never knew about the squirrel angle, thank you!
This may be a hint that you don't want to colocate where I stuff my machines ;)
EV1 was pretty reliable right up to the point that it wasn't. Taught me some good lessons about multi-DC availability. Some of which were recently re-learned by the folks at OVH.
Actually, I apologize that was a huge presumption.
When _I_ am in a sauna I'm normally not fully clothed and running around in a stress induced panic.
Also 60 degrees Celsius would mean it made an appearance here:
https://en.m.wikipedia.org/wiki/Highest_temperature_recorded....
Given that the highest recorded air temperature on earth is 56.7 degrees, you may wish to revise your last sentence.
I was thinking 40 degrees. Given "average" isn't implied by "really hot", I'd say your 40 degree figure falls in line with what I had in mind. But you're right that it's still a big jump to 60 degrees.
Maybe I shouldn't have put vague comparisons in my post because it's rather distracting from the core point that the GP got their Celsius and Fahrenheit mixed up.
https://www.bbc.co.uk/news/magazine-10912578
But it seems like 175F (80C) would be a normal-ish range. temperature.
Probably people die in saunas all the time in countries where it's popular, like they sometimes do when eating, walking, or sleeping or watching tv.
80 degrees C would generally be the hotter end of spectrum. Some go as low as 40 degrees C
When outside temperature is equal or higher than body temperature the only way to expel heat is through sweating. For this the air must by dry enough (wet bulb temperature less than body temperature). https://en.wikipedia.org/wiki/Wet-bulb_temperature
If body can't expel enough heat it results in eventual death.
No Celsius. Admittedly 140 C is very hot (but not unheard of) for a sauna and you should probably not be in for more than 2-3 minutes max at a time. 90-110 C however is pretty standard and really nice. I can highly recommend giving it a try.
https://www.sunrisespecialty.com/sauna-temperature
https://www.saunasamurai.com/sauna-temperature-how-hot-is-sa...
Given someone died at 110 degrees C in a world championship, I'd wager 140 isn't "hot but not unheard of" and 110 isn't "pretty standard" like you claimed.
Perhaps not the PCB itself, but there's typically lots of thin wire around with thin insulation.
A lot of DCs after 2000 went from cold to warm based on the failure rates not changing.
If you get a chance to tour HE Fremont (the old Apple factory), it's not cold.
Factoid: most startups in SV started with a rack in a closet, that one day lost air circulation or cooling and went to the boiling point. I recommend people monitor the temp and cut 2 inches off the bottom and top of their closet door to allow passive cooling while you drive back to the office.
The environmental conditions within the affected Availability Zone have now returned to normal levels. We will provide further details on the root cause in a subsequent posts, but *can confirm that there was no fire* within the facility.
While we lack any evidence on which to base an assertion, The Register has reported on erupting UPSes and tiny puffs of smoke leading to hypoxic gas being released into data centres.
The whole point of hypoxic gas release into data centres is to deprive fires of oxygen. And as humans need oxygen, it can be a while before engineers can return to a data centre.
The Register mentions this as it fits the facts offered in this incident, and with AWS’s language about “environmental conditions” preventing entry.
The Register says that it'll update its article if it gets further info.
[1] https://www.theregister.com/2021/06/11/aws_eu_central_1_inci...
It is 3AM in Germany, and I'm tired and I don't want to know what PDT is.
OTOH, they could add translation on website for it...
AFAIK most maps I've seen have the longitude 180 degrees West at the left edge, and 180 degrees East on the right.
[1] https://www.quora.com/Why-isnt-our-world-map-made-to-look-li...
I weren't aware of that
It's weird af that I've never saw map where America/Asia is in the center
Whilst in hotel quarantine I hacked a webapp together to show timezones around the world.
I did it glance at my remote colleagues local time.
Reminds me of the meeting planner by timeanddate.com (which I use a lot of)