Hacker News Reader

Comment by wpietri | Hacker News Reader

People are responsible for their actions. There were many things wrong with this project, but one of them was fatally bad code, which this guy took money to write. Having him be named strikes me as the very minimum in accountability.

I think it's also important to name him, to interview him. To understand how he came to kill. So that anybody writing life-critical code today can say, "I'd better not end up infamous like that guy. I can't make the same mistakes."

Maybe if he had been held accountable, the programmers at Uber wouldn't have been lax enough to code up the negligent homicide of Elaine Herzberg.

View on HN

> There were many things wrong with this project, but one of them was fatally bad code, which this guy took money to write.

If I remember correctly, the bug in this case was a race condition between "normally" running code and an interrupt handler. The race was only triggered if an interrupt happened in just the right window between two instructions. I'd be willing to bet that 99% of programmers, if simply given the code and asked, "Is there a bug here?" would have answered "No".

Should people writing operating-system-level medical equipment software be required to have a basic training about race conditions and how to prevent them? Yes. Is it fair to expect a random engineer who was ordered by his company to rewrite the code to work without a hardware interlock to know what he doesn't know? No.

> Is it fair to expect a random engineer who was ordered by his company to rewrite the code to work without a hardware interlock to know what he doesn't know? No.

Looking back at when I did Safety Critical Systems at Uni way back in another era, the most important points that we got out of the Therac-25 case study were not to do with bugs at all, but to do with the deficiencies of the system architecture and methodology, especially the decisions that led up to it.

It is 100% fair to expect people working on life-critical systems to either a) know that they are competent to work on life-critical code, or b) to not work on life-critical code.

The guy wasn't ordered to do this at the point of a gun. His bosses asked him to do something, he said yes because he liked the money, and people died. The bosses are also responsible, but that doesn't mean that his negligence didn't kill people.

Is said "random engineer" an engineer? No. He's a code monkey and should be expected to be paid like a code monkey.

Obviously we don't know who wrote the code, so we can't ask him, but did he even know the consequences of what he was writing? Was he even aware of the potential lethality?

The very fact that we don't know who wrote the code is exactly why it doesn't matter. The complete lack of traceability and accountability is what caused these deaths.

There's so many things that went wrong in order for Therac 25 to kill people that it's irrelevant who wrote the code. It could've just as easily been you or me.

> Was he even aware of the potential lethality?

It could be worse than that. He might not have even known the actual purpose and complete function of the system for which he was writing code.

That's common when big projects get outsourced. One might call it "decomposing the problem into small tasks" but it ends up with people writing code for machinery that they have no knowledge of. I've seen this myself. These kinds of projects depend upon layers and layers of oversight and process.

I think that the folks who are looking for "a perpetrator" are utterly missing the key-value of studying the Therac-25 case study.

I am personally not looking for "a perpetrator". Many people share responsibility for a failure like this. But that's all the more reason that we should clearly describe who did what, when, and why.

Please note that you are working very hard to make up a fantasy situation that entirely exonerates the person like you. Your fantasy does not match the known facts. Think hard about why you think that's the most important thing for you to do here.

He knew or should have known the potential consequences of the code he was writing. Maybe this could have been you, but it definitely wouldn't have been me, because I would not have committed the first line of code until I was reasonably sure that a) I was not going to kill somebody accidentally, and b) the project had sufficient safeguards.

I'd also like you to note that you're pointing out the lack of accountability as a problem, and then dismissing one of the fundamental mechanisms of accountability: naming people who harm others.

Yes, other people on the project also should have pushed for traceability and accountability. But responsibility is not zero sum. "I was just following orders" is never an excuse for killing people.

A culture of blame leads to programmers of safety critical software hiding their mistakes.

On a project? Sure. I'm a big fan of blameless retrospectives as a way to maximize quality. But that only works in the context of a quality-oriented culture that is strongly focused on continually improving safety.

However, a) none of that was true here, and b) in the wider societal frame, I think blamenessless must not always trump accountability. When we're talking about egregious negligence leading to death, I think naming the culprits is the very least we should do.

I can see why you would say that, but isn't classical engineering a counter example? Professional engineers have a culture of accountability and responsibility. No system is perfect, but (to my eyes at least) classical engineering works safely without a culture of blame.

Preferable to the "mistakes were made" fog bank of diffuse nonblame. People respond to incentives whether cash or heads on pikes.

A culture of refusing responsibility is better how?

The opposite of a blame culture is not a culture of refusing responsibility, but a system where you look for problems in the process when a fault makes it into the final product. It's always a mixture of specification, implementation, and QA that makes mistakes possible. Putting the blame on the engineer is not very helpful, it does little to prevent future problems. Trying to figure out how it was possible that an implementation bug made it through QA allows you to prevent similar mistakes in the future (at least if you're lucky).

Just to be clear, I'm not putting the blame on the engineer. Many people are responsible for this failure. What I'm saying is that the engineer is one of the responsible people, and that as fellow engineers we should be especially concerned with making sure that responsibility is correctly performed and has appropriate accountability.

But why you want to keep him accountable and not the ones who did the testing and released the product?

I never said he should be the only one. Responsibility is not zero sum. But as a professional software developer, I am pointing out the professional responsibility we all have.

IMHO whoever decided to let out the hardware protection, present in older machines, is the 'culprit'.

Reply on news.ycombinator.com