Hacker News Reader

Comment by peterabbitcook | Hacker News Reader

A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.

I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.

In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.

View on HN

Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.

The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.

A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.

>They are usually on a separate Network.

Then someone plugs in a cable because boss wants something "over there" and there's already a network that runs "over there". Or optimizes to a smart switch with vlans, and then someone else optimizes to a single vlan. It's not hard to not give a shit, or not understand, network security.

That someone can be brought into an office and shown a small diagram of the approved network topology. Then they can be shown a small diagram of the current network topology. Next, they can be asked if they're the same. If they're not, they can finally be asked if they're aware that deviating from the approved network topology without consulting infosec is grounds for termination of their employment.

Bunch of assumptions about operational excellence in there. Doesn't match my experience but, it does match my desire.

You have to be the change you want to see in the world.

We often forget this, but computer software and hardware are engineering disciplines. Part of engineering is knowing when something's bad and telling the person who asked you to do that something that you won't do it. In the civil engineering world, this means you might have to just walk away from the project and blow a whistle before people get hurt or killed. It's time to normalize that in IT.

Great advice for a much younger person.

Or anyone.

It doesn't have to wait 20 years for someone to come through engineering school to happen.

You're assuming that it'll be noticed at all, and that the person noticing cares enough about it to make a big deal out of it - likely involving several layers of management.

In reality it'll likely first be noticed ten years down the line, by someone who lets out a big sigh, mutters something about "incompetent dipshits not updating documentation", and moves on with their day.

I'm not assuming anything.

I'm saying that's what you do in order to solve the issue. You have to actually try, and you have to do actual engineering.

If the local planning commission submits a call for proposals for a bridge to cross a 400 foot chasm over sharp rocks, and they insist that it absolutely, positively must be made out of popsicle sticks, local civil engineering firms aren't going to take up the project, because that's insane.

Why do we give the management of these places a pass for PLC and SCADA systems that could give massive problems - up to and including the loss of human life - if they're hacked?

We're talking about the military. Many years ago I heard a presentation by an IT guy in the marines. He stated that senior officers would regularly give him instructions that would violate some policy or other - such as giving their secure laptop direct access to the internet so they could check their personal email - as an order. That is, they could not refuse. I hope things have changed, but this fellow was dead serious at the time.

Report that shit to your security officer

Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.

also you can't pin the user/pw to the machine with a note, because someone might need remote access. better stick with admin/admin

That’s a tough question. If your PLC is on an airgapped LAN, admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.

On my project the service I wrote was doing bidirectional communication with the PLC over OPCUA. The server running this pod was connected to the internet, so it was critical to have proper TLS for the OPCUA client/server. Rotating LetsEncrypt certs on the system every 45 days is a lot of toil, but using a self-signed cert that expires in 2040 from some dev laptop doesn’t pass muster in most organizations either.

That’s just the OPCUA path.. In these projects You also typically see WinCC HMIs that can talk to the PLC from anywhere on the network without TLS. And also SIMATIC Web Server pages - minimally secured by default, and the amount of info you can grok about the system in Chrome Dev Tools is troubling.

> Rotating LetsEncrypt certs on the system every 45 days is a lot of toil

What is unique about your system using LetsEncrypt that you can't automate certbot to handle this task as it was designed and intended to be done?

On an airgapped system that is is turned on once and needs to keep running for many, many years? Industrial equipment is a world of its own and internet best practices just don't transfer directly.

Some PLCs run extremely expensive machines. Some machines can't afford to have their control systems stutter or fail because that can lead to physical damage and production outages of enormous proportions. A PLC that stops communicating because a certificate just expired is absolutely not acceptable in some plants.

Running certbot on your web server is easy.

Running certbot on a random PLC isn't happening.

I'm not sure if you're speaking from personal experience, but most I've interacted with don't have to worry about the self-signed vs. LetsEncrypt debate. They just don't do it. Also there would be no way to do LetsEncrypt as the system is air gapped.

You can do DNS challenges for air gapped networks as long as the TXT records resolve publicly.

So then you have a signed CSR right? How would you get the certificate onto the box?

> admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.

admin/admin is on a post-it stuck to the monitor because everyone we hire is perfectly capable of rooting a machine they have physical access to.

Yeah, but deliberately rooting a machine you have no business touching means you are instantly getting fired.

Having an actual password prevents people "borrowing" a key to the equipment broom closet from "optimizing" some config values they really shouldn't be messing around with.

Just my opinion but I believe a self-signed is more secure than Let’s encrypt. When a self-signed cert changes and no one in the org knows why that is alarming. Getting certificate error messages every month and half every numbs the org to the alarm.

My guess would be something like the CA using some feature that was newer than what the equipment would support (e.g. ECC signatures but the equipment only supporting RSA), not an intentional "no real CAs" decision.

My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.

Well, what else can it do, really? It has to boot with pretty much zero knowledge about the external world (maybe except asking the user for the current date and time). Sure, you can hardcode an outdated list of CAs (it's always outdated because the system can be booted 10/20/100 years after it was made) in but that just opens you to unexpected obsolescence, and you usually can't put too much stuff in the bootloader anyway.

Not really dissimilar from the human upbringing: leave a baby with "bad" guardians, and it will grow up corrupted. That's a feature, not a bug: if you knew what behaviour exactly you wanted (other than "whatever Simon says"), you would just bake it in in the first place, right?

If we pretend we're revising TFTP boot in 1995, let's have it get up to 20 boot options from the server and their md5 hashes, and if it's not set to auto it waits for the user to pick one. It then verifies the hash as it downloads. Also it uses TCP for the download.

Nah, you won't sell the cow like that. You need to add more reliance on the public CA infrastructure and third-party code signing. Also, "ask user"? Ask the TPM instead — I mean, why would the computer's owner trust the computer's user, right?

> In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.

Stuxnet was over a decade ago.

There should be a simple rule that everyone with the ability to understand things like PLCs should be able to grasp: your equipment does not touch the internet or external storage, period.

Those who can't grasp this concept should be shown the door with a recommendation that they find a less mentally-taxing line of work.

The Stuxnet PLCs weren't touching either, though.

The worm was designed to jump across network air gaps via USB flash drives, and spread across isolated networks to other hosts. Finally, it targeted what was likely going to be a service technician laptop, which had to connect to the PLC via a data cable to update and adjust its configuration.

An attack like this can only be avoided if you never transfer any data from the outside to the inside. But that means you won't ever be able to install any form of software update or upload new config files without manually typing them in - or even reinstall the OS on any machine...

Oh I wish that was possible.

Fact is if the customer wants to put their worksite management on AWS, you inevitably expose the OPC server and/or PLC to a Intranet. This is then inevitably connected to the internet in some way. The customer may or may not put barriers between them, but that's not up to us to decide.

This becomes especially a problem if they have multiple sites across a country or countries.

We have a customer that runs a factory on god forsaken WIFI, then dares to complain about a bad connection to said equipment. Thankfully, they have their own internal IT department.

I work with logistics automation software. Most PLCs I've worked with are legacy and don't even support TLS. Doesn't surprise me at all. I honestly am baffled how old some of these sites are and never got a security upgrade.

Now with the EU cyber resiliency act things are moving but I think it's infeasible for many sites to meet modern security standards, because of both hardware and software.

Reply on news.ycombinator.com