Hacker News Reader

Comment by jakeydus | Hacker News Reader

I work in SCADA development, and believe me, there's a reason the industry moves slowly.

This slowness also means it sometimes has a hard time understand new risks and a lot of industrial systems are not operated by people who understand those risks and do…very stupid things.

100%; I've worked on many a project that was implemented without proper concern for cybersecurity. It seems like this hack in France was triggered via a missed update (according to the manufacturer of the system, who I am hesitant to believe tbh). Security updates aren't always done automatically, and if a system's implemented by a consultant and the owners of said system don't know how to operate it, then you wind up with a bunch of spoiled food across however many supermarkets.

The best systems are implemented either in-house or by a team of consultants working hand-in-glove with the in-house IT or ops teams, then maintained in-house. That's been my experience. Unfortunately your average municipal wastewater plant is probably run on decades-old systems by the same IT dept. that fixes the city council's printer.

This comment could be a lot more interesting, useful and worth typing than it is. :(

Sorry, it was a busy day. I was agreeing with everyone else here. The move fast and break things attitude is one we can't afford when building control systems for critical infrastructure. I used to work in big tech, and if I pushed an update that didn't work or wasn't tested properly, the worst thing that could happen was we revert the change and there's a minor inconvenience to a user. If Facebook goes down for an hour, the company loses money, some advertisers get mad, and their users get frustrated. If an industrial Siemens SCADA operation goes down for an hour, the best case is the operation (be it a factory, airport, wastewater plant, etc.) shuts down for an hour. The worst case is people get sick or die.

It sounds dramatic but every update or change to the system rocks the boat and exposes it to a whole new tranche of dangers. So we spend most of our time observing. There's no such thing as a casual QoL update in SCADA.

What I don’t get is, if it’s so critical, why wasn’t the bare minimum done to isolate the hardware from the Internet At Large?

I think for the big players, it often is, my impression is the more systemic risk is smaller players (like very local utilities) these are sometimes run by very unserious people.

Most SCADA systems date from the 1970s and a lot of them don’t really think about network security or even have a concept of it. So, they don’t think about the consequences of connecting it to the Internet for remote access. It’s just a feature they want.

> There's no such thing as a casual QoL update in SCADA.

> So, they don’t think about the consequences of connecting it to the Internet for remote access. It’s just a feature they want.

This is the disconnect that's so confusing.

I don't work with many of the smaller players, but you're right. They're far more likely to not have their own teams building and managing these systems, so they're susceptible to the Lyle Lanleys of the industry who come in and show them how they can control everything from their phone. They get sold off-the-shelf products and told that they're plug-and-play, but they're usually not.

It’s different sides of the house.

Operators understand how important having them connect to PLCs and various bits because failure can mean you can take out parts of whatever it’s controlling, the power grid, oil pipelines, etc. So, you need to be very aware of what you are doing when you do these upgrades. There’s a lot of process to ensure you don’t break things.

But, you tell them about network security and or having to have Modbus messages wrapped in TLS and some of them will tell you to shut up and get out of their way. These systems have literally been just ported from the equivalent of a PDP/11 to Linux. So, not the most robust things when you come from a network security perspective. They really are 1970s systems and it’s really annoying, but I don’t think there’s any money in really doing anything about it and a lot of people make good money consulting around for these things.

> believe me, there's a reason the industry moves slowly.

No profit in security/dependency upgrades. Propriety SCADA protocols to enforce full stack manufacturer lock in.

That too ¯\_(ツ)_/¯ then you end up with a bunch of civil engineers who build a monitoring and control platform in Power BI and expose everything to the open internet, which then gets exploited, and they wind up working on a Siemens product anyway (I don't work for Siemens, I do custom development). It's really frustrating to watch.

Go on.

Reply on news.ycombinator.com