Tech failure nearly caused massive flood in Amsterdam | Hacker News Reader

Tech failure nearly caused massive flood in Amsterdam

nltimes.nl

74 pointsbelter50 comments

> The department also stationed a contractor at the IJmuiden complex 24/7 so that they can intervene immediately if anything goes wrong.

I'm surprised something as vital for the Netherlands as the IJmuiden sluice (also the largest in the world) wasn't staffed 24/7 until then.

Even if it's supposed to be reliable and automated, and having someone on watch 24/7 is somewhat expensive, it's nothing compared to the potential damage of a malfunction.

Risk management. Numbers game. If it could be insured, I'd think that 24/7 staffing would bring down the premiums.

Why, nothing happened. Let nature run it course. After the tide the water will flow back, and the 32 cm rise is nothing to worry about (after 4 hours, imagine the 24/7 guy observing the 1 mm high wave).

It would make more sense when water levels are already high. But then there are indeed people assigned to keep an eye on the crucial infrastructures .

Where does 32cm come in? Tide on Dutch North Sea coast is around 2m

https://www.tide-forecast.com/locations/IJmuiden-Netherlands...

The sea is a bit wider than the river. And deeper as well. The 32 cm was the rise mentioned in the article.

> imagine the 24/7 guy observing the 1 mm high wave

It's not the flow itself that matters here, the rise is how they found out that there was a problem. The problem is that the sluice was set to manual operation instead of automatic, for an unknown reason, and stayed open when it should not have.

The guy would not have noticed a rise, what he would have noticed is the sluice staying wide open for a long time.

Also, the water rose 32cm in the canal, not in Amsterdam center. The locks were closed in time to prevent water from rising in Amsterdam, where 32cm could have been a problem. The water level is supposed to be mostly constant in the city center, it doesn't follow the tides.

The water does not necessarily "flow back" after a tide either, since most of Amsterdam is below the level of the sea at low tide. It must be pumped out.

Uhm, Holland is here and there below sea level but the rivers aren't. Escher probably could have drawn a map where this could happen, but in reality we still need the rivers to be higher than the sea.

edit: canals and baby rivers are lower indeed, they are pumped out and locked

Glad that this was caught in time.

Perhaps the eagle-eyed Rijkswaterstaat employee should receive an honorarium from the insurance companies and Gemeente Amsterdam.

Also “Operating staff also received extra training.” - just like a standard SaaS major incident wrap up…

Are Insurance companies selling flood insurance in the Netherlands? Must be expensive....

If you have a house you’ll most of the time are required by the bank (because mortgage) to have what’s called an “opstalverzekering” which is what insures damages to the house. These have a component that insures against flood damage but there are probably some limits to it.

But yes, flood damage is typically insured. Premium will vary ofc if your house is inside or outside a protected area.

Not too bad, since the risks are well known and understood. And unlike in the US the government actively has to maintain that risk level by law.

My private insurance covers flooding at no additional cost and almost all home insurance covers it (maximums/deductibles are affected by insurance price of course but they are very reasonable).

I visited the supercomputer centre at UEindhoven back in the 1990s. All the hardware was up on the first floor, which of course meant serious load bearing construction. But this is why.

Why though? Eindhoven is about 17 meters above sea level, and the Dommel seems hardly big enough to pose a serious flood risk. (It obviously does flood every once in a while, but usually not near the university area)

Ironically enough the parts that flood the most in the Netherlands are the parts which are not below sea level. Just this week Doetinchem experienced some high water due to heavy rainfall [1]. In the south-east and east they sometimes also have problems with the amount of water the rivers over there have to process. In both cases the problem is the river can't process the water fast enough.

The parts below sea-level are mostly all in the delta area. Meaning there is little risk of rivers overflowing since they are at their widest in the delta. And as for rainfall; That just means pump water harder out of the "tub" then normal. As for the threat of sea-water in these areas, that's were the Delta Werken[2] are for.

[1] https://nos.nl/artikel/2535587-noodweer-leidt-tot-overlast-w...

[2] https://en.wikipedia.org/wiki/Delta_Works

For North Americans reading this, the "first floor" is not the ground floor in the Netherlands (where Eindhoven is)-- they count floors from zero.

Not just the Netherlands, most of Europe counts floors like this: ground, first, second, etc.

> Not just the Netherlands, most of Europe counts floors like this: ground, first, second, etc.

I think it wildly differs all around Europe.

In Spain for example, if someone says "1st floor" it can be two or three floors above the actual ground floor, if there is a "Entresuelo" or "Principal", and you start counting after those. Actual ground floor is "bajo".

On the other hand you have the "atico" (attic) which is the top level floor, unless there is a "sobre atico" ("above the attic"), so just because you live in the attic doesn't mean you live on the top floor.

Then every region can have their own convention, or even difference in neighborhoods in the same city.

They count ground, first in the country in Europe where GP is from. In turn they generalize based on nothing to most-of-Europe.

Call it the Ugly European effect. https://blog.codinghorror.com/the-ugly-american-programmer/

and Australia...

I'm not sure how many countries in Europe count like that, the online information is totally unreliable. For example: https://www.reddit.com/r/MapPorn/comments/mvnkja/floor_numbe... That map is absolutely incorrect for a bunch of countries, e.g. the Nordic ones.

Sure, but not everywhere, and I was just addressing the specific example.

I've heard even some programmers like 0-based indexing. Crazy!

In Dutch we say “verdieping” which roughly interprets as an additional floor above the ground floor.

It originally referred to attics with deeper floors:

https://onzetaal.nl/taalloket/verdieping-verhoging

I would say it's not counting from zero but it's a "ground floor" (that's basically were your door is) and the floor above is the first floor. There's no "zeroeth floor".

The us must be against this, I was just in a us building where the basement was -1 on the elevator and the main entryway one floor up was 2…

Original source (full 160 page RCA), but in Dutch: https://open.rijkswaterstaat.nl/zoeken/@275817/evaluatie-inc...

Google Translate says:

- they don't actually know what triggered the switch into manual mode

"RWS - Programs, Projects and Maintenance (hereinafter: PPO) has conducted research done, and had done, to the technical cause for the switch to manual mode. PPO has not yet been able to determine the cause. The reason for this is that the logging circuits in the system approximately 1 x per second registers. Switchings can take place in approximately 200 ms according to PPO. This makes it impossible to determine which circuits led to the failure."

- their SCADA in general seems poorly maintained:

"According to the maintenance contractor, there are approximately 60 faults active in SCADA in normal operation"

"In the past, a software issue was found in the vibration measurement system of the pumping station pumps. This caused a technical safety function to be disabled."

"During work on the installation of pump 7 in the pumping station, various software errors were found by the maintenance contractor."

- and operation degenerated into treating it as arcane, poorly understood black box:

"In the event of failure of regular operation, the sluice and pumping station complex can be operated from Schellingwoude using the emergency panel. It is important that this system is used in an emergency. During the incident, this was not used, because operators did not trust the functioning of the panel and were not well-acquainted with the operation of the panel."

- and of course the UX (or Operator Experience? OX?) is poor too:

"In SCADA, the sluice gates are light grey instead of dark grey when the sluice complex is switched to manual operation. However, the light grey sluice gates also occur in various phases of regular operations."

Obvious caveats regarding correctness of Google Translate apply; I do not speak Dutch. Most of this is from page 14 (or page 11 of the "INTERGO" sub-embedded report). And I'm focusing on the negative here — Amsterdam didn't flood, so there was (even if just barely) enough "buffer" in their processes to prevent the worst.

If you want to see how much of an oopsey it was: https://www.google.com/maps/search/IJmuiden+sluice/@52.39235...

The article is only 5 paragraphs long, yet the similarities to so many other "the system was complex, the operators got confused, and things went wrong from there ..." disaster stories are obvious.

I wouldn't even be surprised if that systemic issue was known already via smaller incidents and at least one person who pointed that out but got shoved to the wayside because "too expensive" or similar nonsense that is being thrown at them.

Except there was no disaster, the article says exactly what part of the system failed and what measures saved the day and what changes were implemented.

Literally true.

But from a control systems engineering PoV - the difference between this story and "real" disaster stories is that (in this one) an alert live human realized what the true situation was, and initiated corrective action.

Excursions (from normal behavior / limits) and incursions (beyond established parameters) are warnings of potential disaster, both terms used in various contexts. There's a distribution of such events, usually following a power law in terms of scale or magnitude, and an increased rate of excursions/incursions is often a warning of systemic and procedural weaknesses which increase the risk of a catastrophic event.

Amsterdam came within centimetres and/or hours of a tremendous crisis. Cutting that far into safety tolerances is an exceedingly bad sign.

I recently re-read Saint-Exupery's Wind, Sand, and Stars for the first time in many decades. Following a long career in what I've come to view as technical risk management (systems, network, data, and software administration and/or development), what struck me most wasn't the poetry and beauty of the book, but all the damned stupid risks the narrator took, all of which were warnings that an inevitable fatal accident would occur. And in fact did.

Systems be systems, don’t they?

> water flowed toward Amsterdam unnoticed

I'm glad we in tech are not the only ones that deal with these kind of monitoring issues.

It wasn’t unnoticed at all. Basements and streets flooded, water up to the windows.

And good luck with your insurance. Although now they should at least be same to claim it from the gov

I think you misinterpreted the photo in the article. The photo is of a canal with adjacent houses.

The Jordan was flooded. I think the Anjelierstraat. The relatively new buildings of the Marnixstraat survived the water that was up to the windows

I don't know these places. The parent comment talked about flooded streets which from reading the article does not seem to have happened. Do you think the article misrepresents that situation or am I missing something?

When I first saw the headline my brain autocorrected to "Massive flood nearly caused tech failure...", as that was something I found much more intuitive at a glance. This piece of news is such a great example of "Man bites dog"!

Would a ~30 cm increase cause that much problems? There's plenty of playroom in Amsterdam's canals, the water level is at least a meter, usually more below the canal sides.

It didn't but the margin was nearly exhausted and that's the scary part.

It causes huge problems for the freshwater fish. Even regular use of the Ijmuiden sluices lets enough salt water leak into the North Sea canal that it causes problems for which technical mitigations are being developed.

It was this day: https://youtu.be/kR6PT_yw9m0?si=nOX06CXrdPrmeQNW at 0:42

Pretty interesting to hear that there are no less than 14 other gates and barriers between Ijmuiden and Amsterdam that all got closed as soon as the problem was discovered.

Is there a way for general public to see the status of open sluice gates and water levels in various parts of netherlands? A live datastream might be the best!

I guess 99% of the population would not be qualified to tell if an open sluice gate is problematic under certain circumstances. If it were as easy as "Sluice gate open && water level > X" I am 100% sure there would already be an automation for it.

Rijkswaterstaat publishes quite a lot of information online. Water levels can be seen on a map[1] on waterinfo.rws.nl. It also offers access to historic data in CSV format, but that is handled through a wizard and you get the data by email apparently.

Information about sluices, bridges, etc. can be found on vaarweginformatie.nl, including the live status of many of them. The Ijmuiden sluice complex doesn't have a live status it seems. See [2] for the map.

[1]: https://waterinfo.rws.nl/#/publiek/waterhoogte [2]: https://www.vaarweginformatie.nl/frp/main/#/geo/map

That was on 2023: any reason as why it is reported only now ?

The final report looking into it was published yesterday

Reply on news.ycombinator.com