Microsoft 365 outage leads to chlorine gas poisoning of 15 hotel guests
www1.wdr.de
www1.wdr.de
These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.
First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.
Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.
Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)
Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.
Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isn’t their job. But the staff should notice.
Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too. (In fact, the system I’m familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)
—- sensors —-
Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).
Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.
There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.
Isn't it the chloramines that are smelly? Or at least non free chlorine. A strong chlorine smell is often a sign of a poorly cared for pool or spa
4/5
EDIT: I want to add that if you breathe that crap long enough, it really makes everything else smell really weird/awful/chlorine-y. Until now I never thought to look it up, but the after effects of being around chlorine for a while leaves a distinct olfactory residue, and it is awful.
If you breathe chlorine long enough... won't much worse things happen to you?
You reminded me of nothing so much as The ICI Song:
And it's go, boys, go
They'll time your every breath
And every day you're in this place you're two days nearer death
But you go
There's overtime and bonus opportunities galore
The young men like their money and they all come back for more
But soon you're knockin' on and you look older than you should
For every bob made on the job, you pay with flesh and blood
Plain chlorine ought to be noticeable before it sends people to the hospital.
One would have thought so, the story itself doesn't add up to me[1], perhaps it might be a translation issue?
Gaseous chlorine[2] systems for places like pools and hotels aren't all that common (I couldn't think of anywhere outside civil water treatment or major industrial), liquid chlorine (sodium hypochlorite [3]) chlorination systems are a lot more common because they're supposed to be relatively safer to operate.
Sodium hypochlorite is certainly much safer to handle compared to straight chlorine gas[4], but it will still give off the characteristic chlorine odour and will still happily hurt anyone not respecting it.
I wonder whether it was an absence of mind thing for patrons if the pool was indoor, the smell of chlorine tends to be stronger than outdoor pools? (enclosed setting, higher dosing given water warmth, etc.)
[1] Hazmat trained firefighter (though I am not a chemist)
[2] https://docs.airliquide.com.au/msdsau/AL640.pdf
[3] https://hyclor.com.au/downloads/msds-sheets/HY-CLOR_Liquid_C...
[4] Both have similar response requirements (Hazchem codes 2X, vs. 2XE), though chlorine gas releases demand much more extensive public safety measures (evacuation). See UN 3310 and 1791: https://www.ntc.gov.au/sites/default/files/assets/files/Aust...
The pool in the article clearly needed more than 30 ppm, they definitely did not sufficiently cover their asses here.
CYA stands for cyanuric acid, not cover your ass :) It’s added because it reversibly binds to chlorine. This causes the majority of the chlorine in the water to be in a mostly inert state in which it’s not irritating, doesn’t offgas and is protected from UV. You need very little active chlorine to control bacteria, viruses and algae, but you need considerably more chlorine to avoid the water becoming locally dechlorinated due to any sort of contamination that consumes chlorine. CYA helps fix this mismatch.
If I were to guess, this is not the only problem that hotel has as it speaks of systemic issues.
I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.
You know what they all have? An email account
Wtf?
I must have missed the memo where Slack Discord or “apps“ became suitable for safety critical task. Must be what USA has tried to use just before grounding all planes.
1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.
2. Could Microsoft have not had an outage? Hard to fix.
3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.
And you have to check those lights and sirens that they still work.
Everybody knew email was down.
The real problem is nobody thought of the consequences regarding the alarm system
Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.
It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.
EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.
Testing requires staff and you can be sure those parks are understaffed.
Best thing is to avoid them, next best thing is a cheap solution with low maintenance by the staff to notify as much people as possible.
Email wins.
Not a good solution but the best you will get for the price they are willing to pay
Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.
And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.
The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.
Curious: what do you think has to be done with an email?
Sarcasm aside, in this case, the email should've triggered the literal lights and sirens. Preferably automated (tho in that case, why even go through email as middleman) but could also be an actual person flipping a switch.
You see this in software ops too: certain subject formats trigger a special ringtone or even a pager call. Emails are too subtle even with ALERT ALL CAPS SUBJECT LINES to be alarming.
The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.
> But it is accurate to say that the cause of the issue was the outage
The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.
Not one but two of the hotel's systems failed to operate properly. The chlorine pumping system pumped too much chlorine and the last resort alarm failed to alert.
As I said, there's plenty of blame to go around as a matter of general principle. But for what happened here the hotel bears the responsibility towards their guests, not the pumping or monitoring system developers, not Microsoft. The only reason MS in the title is because that failure is big news these days.
On a technical note, even if they had a backup email provider they'd still be screwed because the people getting the alert were probably employees on the single provider (back to SPOF). SMS could have been a good parallel option. Ideally also some physical control panel that triggers an alarm when any safety critical condition occurs. On the chlorine pumping system, that shouldn't be a dumb preprogrammed pump, it should come with it's own sensors and detection loop independent of any external system, something that proactively stops it before it breaches the safety limits.
Using the unqualified "cause" is misleading click bait and in some places it could get you sued
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
In short, all parties suck here (MS and hotel)
Go invent a transport layer that will never fail, and make yourself rich.
Because right now, there is no such thing.
Email is not.
And yet, still not "guaranteed," which was the criteria specified by the OP.
And yet, still not "guaranteed," which was the criteria specified by the OP.
Yes it's true, no system is guaranteed to function, but for Pete's sake, it's clear to everyone that having only an email-based alarm system for chlorine levels is the wrong thing to do just because "well there's no alarm system that is guaranteed to alert people, so let's rely only on email, tee-hee".
Any redundancy would have been better than this. People's lives were at risk, and here we are, debating the meaning of the word "guaranteed". I'm going to be frank here, are you an MSFT shareholder or tied to the hotel in question?
> the user of the pool should be able to notice that something wrong by them self.
Clearly, their motivations for autonomous self-preservation were insufficient.
Reformulated: If pool management's cue that something's wrong is because their guests are leaving the water, they are already liable for damage.
Similar to doing regular restores of your data backups. If you never test a restore, you don't have a backup.
I did an apprenticeship as an electrical engineer and have been in those catacombs underneath such water park type of things. There should be such devices present, and of course personnel to operate them.
I mean, can you imagine your boss' face when you tell em the reason your entire system is down is you didnt get the email from checkmk that something is wrong? Instead of, you know, checking the dashboard yourself periodically.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.
Email is just not reliable. It could be part of a combined system of other things that are reliable.
We've been through this with smoke detectors, it's not a new thing.
The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.
Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.
More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.
And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.
It just has to be annoying enough that even the most junior employee would radio in their supervisor to get it to stop
People had started sending calendar invitations for important emails, on the basis that recipients would then reply to the email and decline the invitation.
This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.
I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.
And yet, as late as yesterday, the idiots in charge have announced that there is absolutely "No Need To Investigate the FAA Ground Stop".
Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.
When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.
This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).
anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.
"Why did people get gas poisoning"
The point of a process is to constrain incompetent people into doing something that will work despite their incompetence. Five Whys is entirely subjective and can never constrain anyone; it's just something that some people think sounds deep.
yeah, these companies need to be told to stop relying on cloud services for everything.
its wild to me how many critical things that absolutely do not need the internet are relying on the internet.
much blame to go around here:
- whoever designed the stupid alert system.
- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”
- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”
- accountant types and their “we can save pennie’s if we cut all of these corners.” and the stupid corners they choose to cut.
- etc…
on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.
5/5
Regardless of high availability promises of cloud services, a local internet outage will supersede the risk and is a risk factor in most places of the world (probably everywhere). Risk that nobody aside the local ISP could quantify either. So logically any system critically depending on the cloud and isn't designed with an offline fallback has unmitigated risk factors.
Sure, if you host some online service, this risk cannot be mitigated. But it can for everything else. And if you need an offline operation mode, in most cases the non-cloud design is simply superior. So cloud is synonymous with having a single point of failure?
This case is one where on risk analysis was done at all or this would have surfaced pretty quickly. There are problems too if a pool doesn't get chlorinated, but it is probably better than to poison your guests.
Knowing that your email could be delayed by days changes how you view it and the applications you deem it suitable for.
Title: 5 nines of uptime
A grand piano is suspended from the ceiling with an electromagnet.
The electromagnet works as long as there is power.
No special battery backups, anything.
Guests are invited to stand underneath it, after signing a waiver.
That’s it. That’s the art.
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
> And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
"Implied warranty of merchantability" is a term of art in United States law, and it applies only to tangible goods, not to services.
I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?
Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.
I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.
Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.
Maybe a waterpark resort, but hotels no. These days they barely afford a front desk employee and a cleaner.
When an incident like this happened, they might add new "system" to improve the situation, probably in the lines of having someone to patrol and check off a list routinely.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
I’m so sorry.
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
If I built a chlorine tank out of toothpicks and duct tape, and it leaks everywhere and kills people, you don't publish a headline that says DuctTape and Global Toothpicks Inc are responsible for chlorine deaths.
A freezer is an obvious example of something that should be controlled directly by a human operating on it and not by some "smart" connectivity.
* 13 guests complaining about irritation of airways (+2 later)
* 4 taken to hospital, 1 described as medium injury, rest discharged
* Dual failure - dosing system had a defect (dosing too high) and the email of the monitoring failed (M365). Article sounds like it's two distinct systems.
Overall the German text sounds a lot less dramatic than the hn title.
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
> 15 guests injured at Phantasialand
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
> Subject: Fire
Dear Sir/Madam,
Fire! Fire! Help me 123 Carrendon Road. Looking forward to hearing from you.
All the best, Maurice Moss
Like, what leads a teenager to do such things?
Me neither, but that is probably attributable to the fact that I didnt have a swiming pool.
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
> With one experiment, he created chloroform and as the book encouraged him to sniff the chemical, he did so and was unconscious for more than an hour, according to his recollection.
Back in the day, i tried to sniff NH3 because i've heard it smell like piss.
It's a lot of worse.
High chlorine levels tend to reduce chloramines, so gasses like nitrogen trichloride seem unlikely to be the problem. You need a nitrogen source to make any large amount.
You favorite AI will happily model it, and I wouldn’t trust it to get the right answer :)
Really? I would expect to get refused or downgraded for "bio".
This is usually a problem for a hotel/motel too small to have a control room, but big enough to have escalators, elevators, HVAC, pools, laundry equipment, fire suppression, water heaters, and other gear that needs to be monitored. What usually happens is that some manager's office or security guard station has a bunch of unrelated alarm panels. That's where medium-sized problems get lost.
The next time you walk by a fire alarm panel in an office building, look to see if it's displaying any faults.
[1] https://pacificblueengineering.com/advanced-control-system-a...
Actual headline: "15 Gäste im Phantasialand verletzt" (15 guests injured in Phantasialand)
And the browser tab shows the following summary title "Chlor: Phantasialand bekam wegen Outlook nichts mit" (Chlorine: Phantasialand didn't notice anything due to Outlook)
> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.
1 dead, 8 non-fatally injured.
They are dangerous idiots, and no wonder that's a gross incompetence. Even more so, since they stupidly blamed MSFT instead of themselves.
I wonder what else could an audit uncover.
It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.
It should be illegal to build a safety system that depends on someone reading an e-mail. But there's no regulation saying "no safety system should depend solely on someone reading an e-mail." There's no generic regulation covering software used for safety. A few select industries and specific cases have limited regulations; but mostly it's the wild west. So we end up with these tragic, preventable accidents.
Governments need to pass regulation about how software can be used in safety-critical applications. At minimum, software should pass a systems engineering safety review, use standard approved methods, and pass independent lab testing. Every other safety-critical device has to follow these requirements. Software should not get a free pass. You can see now what happens when they do.
This event is just one story. There are thousands more like this, buried in the news, or never reported at all. Nobody notices because software is everywhere. You see the symptom but not the cause. CrowdStrike's shitty software ended up forcing airlines to cancel 7,000 flights, causing $500M in damage. Two days ago, shitty software/systems (related to AI) grounded all flights in the USA. In the UK, the BPO Horizon Scandal caused deaths of dozens and ruined hundreds of lives. Regulation can put processes in place to prevent these things. Without it, the problems will continue; with AI involved, it's going to get worse.
People will continue to be harmed by badly designed, badly tested, unregulated software until we do something. Next time it might be your kids poisoned at a water park; by then it'll be too late to act.
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
Story: early days of Windows, they invented a power-save mode. We updated our furnace controller PC with the lastest. The customer complained - it would stop regulating the furnace in the middle of the night shift, just sit there without moving the electrodes for hours.
Upon investigation we discovered (of course) that it went into power-save mode. That cost the customer, but more interestingly, considering a 30MW furnace ran for 8 hours without producing a result meant that it would have been better to have never invented the 'power saving' ecological feature that year - the savings for the nation was burned up in that 8 hours.
They should have use a highly available Cloud-based Kubernetes-cluster on Talos Linux!
"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."
We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.
Maybe “we” do need to do that.
If only to be aware of expenses.
The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.
This seems like a solved problem, e.g. for smoke detectors.
- External system polls the monitor and records levels.
- Local system reaches out to external system and fetches status.
- Someone has the responsibility to do rounds and record the status indicated by the local system.
External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.
You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.
In reality such system would end up sending email notification because it's just too complex.
I'd hope we can do better tho.
It can be very much in-band, like a fire alarm primarily also is.
Who is in charge of all that???
Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
Microsoft has many other offerings for such things, ranging from some services inside Azure to software (like operating systems) intended for real-time use on on-premises hardware.