Colonial Pipeline precisely does keep it's control network disconnected from the internet - the only thing that was ransomwared is their corporate network. They shut the pipelines down voluntarily to prevent further spread.
Many industrial SCADA systems (nearly all) send data from their "OT" systems (PLC/DCS/SCADA) to their "IT" and business layers (Historians/Timeseries Databases, Dashboards, Power BI/etc). This almost always happens through a two-way link (think TCP/IP, HTTP). While the software should not allow data flow backwards, the hardware absolutely does. So how much do you trust the software?
I often advocate that industrial SCADA systems utilize "data-diodes", one-way opto-isolators, or other physically verifiable methods of confirming that no information/data/instructions can get from a "higher" layer (OSI Pi, PowerBI) to a lower layer (Allen-Bradley PLC, Siemens PLC, Emerson DeltaV DCS, etc).
Convincing the powers-that-be to do this has been incredibly impossible in most places and a large reason why I'm trying to transition to a different space - I simply have had ethical concerns about providing engineering services to critical infrastructure without building in best practices.
Stuxnet was over a decade ago - I don't understand how these protections aren't mandated by the DHS already.
Disclaimer: I don't think it's reasonable for non-involved people to assume the OT side has been compromised. I do think Colonial will need some time to verify the integrity of their SCADA systems and it makes sense to keep the power to the physical devices (valves/pumps) offline until they do. I understand why they chose to shut down but I don't think there's any evidence that they'll be unable to start back up again.
Lastly, I saw a quote in one article:
>>> Digital Shadows thinks the Colonial Pipeline cyber-attack has come about due to the coronavirus pandemic - the rise of engineers remotely accessing control systems for the pipeline from home.
I strongly doubt this. It's possible, of course. But it's extremely unlikely to me that employees would have remotely accessed OT/SCADA systems from home. No one I've worked with has had that capability enabled.
Many companies use products which have been shown to have flaws, like Citrix or various corporate VPNs. These could be compromised to get access "closer" to the OT layers but never directly into it.
Onion layer security is very much practiced everywhere I've been.
Edit: I have heard of some petrochemical facilities moving towards allowing operators and engineers to manipulate valves/pumps on their iPhones. This horrifies me for many reasons. I've never actually seen it implemented and I always bring up Stuxnet when I hear people mention it. I personally believe that DHS should make this sort of thing illegal for critical infrastructure. Many good engineers disagree with me.
That said, determining whether or not a system was compromised can be incredibly difficult. I'm sure they'll face massive pressure to turn the pipeline back on as it does supply almost half of the east coast with oil. I wouldn't want to be the person who has to make that call when it's impossible to prove a negative.
CPC had two explosions a few years back which caused gasoline shortages in new england, that may provide indication of the scale of disruption to expect.
I'm quite surprised and comforted to hear that the leadership there is competent and knows how to manage people. I've heard from friends/acquaintances who have worked in the energy industry about how terribly things are put together on an IT front (PG&E being a prime culprit), so I was expecting the same here.
I really like your "data-diodes" concept. Interested to see if such a thing takes off especially as these attacks evolve.
Or would this be some other kind of physical interface that took some kind of read-only data (serial?) and sent it up the layers using TCP/IP, where only this box would be at risk?
Edit: looks like you answered part of this below — you suggest switching to UDP protocols.
Firewalls are currently used, and probably generally configured well. Petrochemical companies have a many-layered onion security strategy with minimal communication paths through the firewalls. Generally you might have 4-8 layers of firewalls from public facing internet to the PLC/DCS/SCADA. Administrative people might VPN 1-2 layers deep and engineers would at worst get remote access to the historian, 1-2 firewall layers above the PLC/DCS/SCADA.
It's my professional opinion that firewalls are not good enough for critical infrastructure. Even a completely air-gapped system was hacked thoroughly over a decade ago in Iran (See Stuxnet).
Your suggestion would suffice, if that box ("gateway", in the IoT parlance) was connected with a one way physical connection to the SCADA system over serial or what-have-you. Then it could communicate with TCP using existing application stacks.
I am designing a system like this at my current job, where luckily we are a small enough team so people have genuinely listened to my suggestions about this.
However, good engineers often disagree with me. I may be overly zealous on this particular issue and I take a lot of criticism about how dogmatic I am at times. I'm not a senior engineer by any stretch.
SCADA devices are not read-only.
That would be like deploying the landing gear of the airliner, because someone triggered a bug while changing the channel on the in-flight entertainment system.
But a hacker wouldn't be able to use their access to the Timeseries database for supply chain and logistics, to pivot to the SCADA system because their attempts would be blocked by a lack of a physical layer connection in that direction.
It would significantly reduce the attack surface of the OT systems.
Apart from this being technologically impossible, if he would have really done it he would have been charged for endangering an aircraft and prosecuted. (So the only explanation why he isn't in prison is that it didn't happen). The technical reason is Data diodes are common in aviation to separate IFE and CAN bus or position data. (e.g in the ARINC)
[1] https://www.pentestpartners.com/security-blog/a-pen-testers-...
The US government term of art for this pattern is a “guard”, often with a regex or manual filter.
A webcam? Second hand, but that's what I was told a dam operator was using as a "grass-roots" solution a while back
The ones I have worked with convert a TCP stream to UDP, send it across the diode, and then convert it back to TCP. Each UDP packet has a sequence number and there is a single reverse-diode that is fired when a packet is missed or arrives out of order that triggers a retransmission of the last N packets.
Low, fixed-bitrate transfer over unidirectional fiber optics. Unidirectional transcievers are the norm for long-haul fiber.
My local electrical utility is still running 11.52kbps RS-232 over fiber for exactly this reason. At those bitrates you don't need backpressure -- your disk will never fill up and if the CPU can't handle that bitrate you already have much larger problems.
It's kind of funny that they have sheaths where one strand is running this piddly dozen-kilobit protocol and other strands in the same sheath are doing 10gbit/sec * 16-channel CWDM.
Most electrical utilities are into fiber optics in a very big way; they already (usually) own the poles and unlike copper it's nonconductive. Many of them have vastly more strands of fiber between substations than they need.
The inside (isolated) server would poll everything, store it into a buffer, and send that buffer (plus error correction) out through an optoisolator to other server.
The outside (internet facing) server would then keep up with the ring buffer, and serve requests, and do any outbound push of data via any protocol required.
A system to do this could be made with a pair of raspberry pi computers and a little bit of discrete components for less than $150 in hardware costs.
A common example is a SQLServer database of all industrial data that is authorized to share with the enterprise. Grab new and changed data as it arrives on the industrial side. Push unidirectionally to the enterprise side. Insert/update the data in an identical SQLServer. Enterprise users & applications interact normally and bi-directionally with the replica database.
The technology is used routinely to provide access to industrial data that enables business efficiencies, without providing access to the industrial systems that produce the data.
For more info see: https://waterfall-security.com
However, a write-only ROM system is possible as long as the ROM chips were reasonably affordable and a company could provide reasonable turnaround times for small modifications. That would move the target of vulnerability up the supply chain.
Some of the things which matter though are necessarily run-time variables like "is the valve commanded open or closed?" and "what are the tuning parameters for this PID control loop?". It's always theoretically possible for a buffer overflow/rowhammer/etc to flip the bit responsible for the valve's open/closed command. Even with an OS/Application stack burned into ROM. You still need RAM.
At least power cycling a readonly-storage device would remove any malicious RAM changes.
I did say ROMs, but you can also use EEPROMs, which are erasable in-circuit, and you certainly put a physical write-enable in that circuit. Ideally, it would be a momentary push-button that has to be pushed in person on-site.
Back in college we used EPROMs, which are erased by putting them for 20 minutes or so under a UV lamp. EEPROMs came out later.
With a pipeline, if sections operated autonomously but cooperated with each other, and one goes berserk, its neighbors will shut down, but they won't be damaged. The repair work only has to repair the one section.
People driving cars are essentially doing what is best for themselves individually (within the bounds of the law), and that ends up translating to something that works for the whole. With a pipeline, that might not work: If pressure gets too high in one area, it might take highly coordinated control across thousands of miles to bleed off contents into buffer tanks & ease pressure a dozen segments away.
I'm not saying that couldn't be done, I'm sure the SCADA systems could be isolated from each other in this way, it just seems like it would require a lot more difficulty with explicit coordination between technicians, not a self-organizing system such with driving on a highway.
"OT" vs "IT":
"Operational Tech" (pipeline and safety-critical monitor and control)
and
"Information Tech" (payroll, email, other business stuff)
?
I could only imagine trying to tell a large corporation that their "IT" authentication system can't be linked to the access card keys for the front gate, or whatever other physical security they might have in place.
It doesn't matter if we can formally prove that a remote access system is sufficiently secure as to aloow engineers to operate valves and pumps from home... For inevitably, some months from now, a wildly insecure utility will be connected to that, and you lose the ability to reason about how to keep the streams from crossing.
The government and industry are all talk. Until we see actual enforcement / incentives for secure hardware, just assume everything (and I mean everything) can get shut down at any time. The only people who think this is an exaggeration are those who haven’t seen what things actually look like on the inside.
Can someone please link me to a page that goes into more detail about this secure hardware framework? Not a month from his passing, and we get a national-security-level attack that might have been prevented if US business took more seriously software engineering and security engineering from back then.
Also, I cannot help but wonder if Dan would still be with us if the irretrievably broken US healthcare system was a national system that supplied him with an inexpensive CGM and insulin no matter his employment situation.
I can't speak to non-electrical infrastructure, but the NERC CIP "high impact" standards already make it largely impossible to operate critical electrical infrastructure from anywhere other than a secured control centre. Operating from your laptop or iPhone from the kitchen table is however allowed for "low impact" assets like small power plants.
I used to work in fabs and every couple of years some tool or other would get a virus, sometimes it spread through the network.
You can't eliminate the possibility of malicious action, Stuxnet proves that. It's my opinion that at least for critical infrastructure we can probably make things much more difficult for our adversaries at a relatively low cost. This pipeline is purported to carry half the gasoline/diesel/heating oil to the east coast, but I'd be lying if I said I knew exactly where the cost-benefit equilibrium should land.
I haven't seen any evidence that the "OT" side of their network was compromised in a way that would cause physical damage, a la Stuxnet.
Or worse - rapidly open & close valves in rhythm, and the water hammer effect (the inertia of the petroleum in the pipeline) would cause the pipeline to destroy itself. The repair costs would be astronomical - you'd naturally have to repair the damaged sections, but then also re-test all the welds to see if any had been weakened by the pressure pulses.
Onion-layer security rather than air gaps. Communication through the firewall isn't supposed to allow control over the valves, but it does communicate both ways (TCP/IP). This is the general practice in petrochemicals, at any rate.
Think what would happen if the autopilot suddenly went berserk and did a hardover.
I could imagine a situation where information another network (e.g. orders or incoming flows from another customer or user) is necessary to run the pipeline but unavailable to use to operate the pipeline control system.
>Colonial has not given any public indication as to the reach of the ransomware outbreak, but Robert M. Lee, chief executive of cybersecurity firm Dragos, said he believed Colonial's operations network was shut down proactively "to make sure that nothing spread into those systems."
(To a first approximation, generic == less interesting and specific == more interesting on HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...)
We need to move to using physical layers where data can only be transmitted in one direction (and then use something like UDP)