Hacker News Reader

Comment by TickleSteve | Hacker News Reader

You most certainly don't want people to be able to modify safety critical code within a pacemaker.

What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with.

What is a problem here, is that the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence.

If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors, or simple things like that. Its the design of the system as a whole at fault, and that is already open.

Medical devices such as these are not black boxes to the people that certify them, everything is open to them, source included. Having worked in that sort of area, I trust the systems that are in place.

View on HN

Industrial control systems are also safety critical systems that have to adhere to very similar regulations as medical systems. Yet, when you ask hackers in that field you'll quickly learn that they have terrible code and abysmal security. Rules and regulations do very little to improve code quality and security, imho. Most things in these regulations are either best practices that any software engineer does (write tests, for example, or do code reviews), or just checkboxes for the QA to check.

I'd argue that over time safety critical systems have worse code than normal software because refactoring is almost impossibly expensive due to all the paperwork that each software change ensues.

industrial control != safety critical.

"industrial control" covers both certified and non-certified code, you will have to be more specific than that for the purposes of this conversation.

Quite often, cases such as the pacemaker are flaws in system design, not code. (e.g. no secure messaging, probably due to the lack of awareness when it was designed).

That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code.

True, the expense and overhead does indeed affect the level of change that is acceptable to a business, but given that change is allowed, the quality of that change is what we're discussing here and all I'm saying is that there is a lot more rigour involved than most developers here are aware of.

> That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code.

I had assumed that as well until all of the horror stories around Toyota's firmware came to light.

https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...

Unfortunately, we are human...

Yes, the toyota case is a well publicised case. Consider though, the number of safety critical systems that are out there performing perfectly everyday. Of course, that is not proof of much, but the fact that you can name the Toyota case (and probably the Therac 25 case) means that the process generally works.

>Rules and regulations do very little to improve code quality and security, imho.

In fact, I'd say the restrict them. Adding a tonne of paperwork is a great way to prevent people from "noticing" problems.

When the paperwork is describing the system design from high level down to low-level to sufficient detail that an external auditor can use it to examine the source then no, that is not true.

The security/safety processes (as applicable) are examined at every level from system-design all the way down to implementation.

Its not perfect, but its best practice we have for secure/safe product development.

If the paperwork was that sufficient then you could compile they paperwork.

that was known as model-driven-development.... been thru that....

Starts with a U and ends with an L? I'm so, so sorry.

Software Freedom Law Centers claims that the FDA requirements for medical devices put most of the responsibility of quality assurance on the manufacturer. It does not seem like the FDA provides much of an assurance of quality. Note: this is not my field of expertise, there is good chance I am very wrong, just following a short paper trail of laws, there maybe more laws (or practices) that I am missing that may indicate different FDA regulation controls for medical devices.

https://www.softwarefreedom.org/resources/2010/transparent-m...

https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFR...

  rules and regulations [...] in a different league than what
  most developers are familiar with. [...] the design (not the
  code) apparently did not take into account any messaging
  security, relying on obscurity as its only defence.

Imagine you were building a suspension bridge to the highest safety standards, and you had people with microscopes manually inspect every grain of sand and cement that went into the foundations. But the suspension cables were made of old washing lines.

You would be achieving a very high and a very low standard of inspection at the same time.

Some people would say standards of inspection are only as strong as their weakest link; that the whole claim of such inspection is to prove the absence of such weak links; and that a standard that has approved such a bridge has, by so doing, shown the claimed proof of safety is a useless joke.

Other people would say the inspection is a checklist of common errors, rather than a proof of the safety of every aspect of the system; that proof of perfect safety was never its claim or goal; and that this merely shows there should be some extra boxes on the checklist, which was in any case a constantly evolving document.

Inspection in this case is a proof that the design you're certifying is fit-for-purpose.

So in your example, it would show that the loading and stresses on your washing lines were sufficiently low to meet the safety margin of the bridge.

Certification of safety-related and medical devices is not a check-box exercise, it looks at the dynamic and static behaviour of your system as a whole (not just the software).

Perhaps I wasn't clear in my analogy.

I mean the washing lines weren't fit for purpose (just like the insecure comms wasn't fit for purpose) but the inspection didn't identify them as unfit for purpose (just like the inspection didn't identify the insecure comms as unfit for purpose)

Given that something unfit for purpose was certified, certification doesn't prove something is fit for purpose.

Of course, certification may indicate something is probably more fit for purpose than something that failed the same certification or didn't attempt it.

The inspection of the washing-line design should have identified this... unless human-failure.

So your point is:

"we don't know what we don't know"?

True. I see no way around this.

Software certification is a joke, and mostly consists of checking off a few boxes on workflow and development methods.

Smart electricity meters are safety critical and I know students writing on it in their summer jobs. Airplanes are safety critical and they accidentally bridged the on plane wifi into the fly by wire system.

Unless this stuff is written in COQ and formally verified, I won't trust it a bit.

Certification most certainly isn't a joke.

It involves a hell of a lot of work, enough to double or triple the software portion of a project.

Whatever certification you have been involved with may have been a joke.

Safety-critical certification most certainly is not.

BTW: I'm s/w architect on a set of smart meters. And they most certainly are not safety critical. (IMO They should be, but thats a separate issue).

...and no Wifi was "accidently" bridged. evidence?

Not Wifi, but IIRC a guy called Chris Roberts was able to leverage a wireline connection to an underseat IFE controller into FADEC control on multiple aircraft. Last I heard of the case, he might be going to prison for it, so apparently at least some people find him credible.

Similarly, witness recent revelations about car hackability - I forget if it was Blackhat or DEFCON where a couple of guys demoed a fully remote attack through the entertainment system's cell network access that culminated in the ability to override the steering, throttle, and brake controls.

Maybe these are not what is meant by "safety critical" in your use of the phrase. If that's so, I think it would clarify matters greatly for you to define what you mean by it that's more specific than "can kill people if it goes wrong".

A few points here:

- Given physical access, all bets are off.

- "safety critical" refers to software that has a SIL level(https://en.wikipedia.org/wiki/Safety_integrity_level). Note the list of IEC/ANSI/EN standards.

- Yes, the hack of the braking and steering system via the entertainment system is a breakdown of the certification system. That should have been found earlier. This is why critical systems are air-gapped. I personally have never worked on automotive systems so am not familiar with the regulations there, but I was/am extremely surprised this was possible.

SIL usually deals with complete systems and deals with probability that the system will deviate from it's designed behavior, whether such designed behavior is actually correct is somewhat orthogonal problem that is often mostly ignored. (same thing applies to attempts to apply formal proofs to software). The wikipedia article lists some of the problems with SIL rating itself, and I've personally seen multiple instances of "running two redundant SIL N systems produces SIL N+1 system", often with the original N being derived by wishful thinking (this seems to be especially prevalent with engineers with railway signalling background, where I accept that for systems based on relays with redundant coils it holds, but it's complete BS for things with non-trivial software).

Various attacks on car ECUs cleanly show what the problem with current relevant certification processes is: each component is certified separately and nobody cares about the complete resulting system (head unit is not safety critical, engine ECU is, but has no untrusted inputs and there is bunch of things in between that are classified as one of these two categories). What is ironic in the automotive case is that the whole reason why there is immense number of separate ECU's in typical car[1] is safety (ie. limiting impact of one ECU completely failing) and safety certification.

[1] my car has separate ECU for each door even though rear doors does not have power windows and central locking uses dedicated wires. I assume that only purpose of said ECU is that diagnostics system can detect when the door is missing.

Interesting points about the automotive world.... and I absolutely agree with you about the certification problems with a mix of safe/non-safe components.

I've never heard the 2xN = 1x(N+1) argument, but will that improve for >2 ? e.g. triply-redundant systems?

I think the root problem there is the "wishful thinking" isn't it?

The 2xN = 1x(N+1) argument is usually presented as being somehow directly derived from IEC 61508, which is certainly non-sense. I've even seen reasoning along the lines of "it runs on Windows NT, thus it's inherently SIL2, there are two redundant IPCs running that code, so it's SIL3".

As for dual-redundant vs. triply-redundant it highly depends on whether shutting the system down on failure of one redundant component is desirable outcome, both railway signaling and most industrial systems can and are designed that way, but for systems where that is not possible (either because they need some non-trivial actions to get into safe state or because the whole SIL dance is about keeping the thing running at all costs for business/legal reasons) the dual-redundancy actually decreases the reliability, because additional components handling the redundancy also have their own failure probability (and in many systems does fail more often than whatever it is supposed to protect from failure, especially in master-slave systems that attempt to detect which of the halves had failed and respond to that by failover to the other one).

Interesting approach that is often used for road traffic signalling and general industrial control is that there is second control system, that only checks that outputs of the primary control system are consistent (for traffic signals, it is trivial boolean function of the outputs, usually implemented in hardwired circuitry) and shuts the whole thing down when they are not.

I'm not sure, but your correspondent may be referring to Hugo Teso's 2013 presentation for Hack in the Box, http://conference.hitb.org/hitbsecconf2013ams/hugo-teso/, or possibly to design revisions of the Boeing 777.

If so:

"The described technique cannot engage or control the aircraft's autopilot system using the FMS or prevent a pilot from overriding the autopilot," the FAA's statement explained. "Therefore, a hacker cannot obtain 'full control of an aircraft' as the technology consultant has claimed."

"The statement went on to explain that although Teso may have been able to exploit aviation software running on a simulator, as he described in his presentation, the same approach wouldn't work on software running on certified flight hardware."

http://www.theregister.co.uk/2013/04/13/faa_debunks_android_...

> everything is open to them, source included

Certifiers typically don't review source code. They are operating at least two levels of abstraction away: they review documents that describe procedures that are supposed to ensure quality.

> If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors

That is a big fat citation needed. Researchers working without access to source code have demonstrated multiple vulnerabilities in safety-critical medical devices. We have no way of knowing how many more they would find if they also had access to the source.

I used to Reverse-Engineer medical equipment (MRIs, CTs, etc). The lack of engineering strictness in theses systems was amazing to me. They are horribly insecure, BY DESIGN

My experience is that, no, this is not an acceptable solution. I have no reason to believe that implanted medical devices will be any better.

Reply on news.ycombinator.com