Hacker News Reader

Comment by chroem- | Hacker News Reader

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when in reality other engineering disciplines deal with unexpected failure modes all the time. The only difference is that these other engineers are motivated to be significantly more thorough about their jobs.

View on HN

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs.

Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that?

We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up.

If we want to hold software developers accountable for downtime, unprompted data leaks, and so on, fine. We should do it proportionate to the damage caused - there's no point in pretending that a shoddy smartphone game is as bad as a shoddy bridge - but fine.

But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd.

>* Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up.*

Even more so, how many engineering problems have fuzzy requirements that on top of that change constantly and arbitrarily?

If someone thinks building bridges is the same as writing some big software, they really have no idea about working in the trenches.

How many software projects have to deal with gravity? Weather patterns? How many software projects need teams of construction workers to actually build?

Different kinds of engineering have different requirements. Just because the challenges in software are different from those in building bridges doesn't mean we get a free pass on correctness.

>How many software projects have to deal with gravity?

Gravity comes with a pretty little equation, and as far as problems goes is as predictable as it comes.

Tolerances of various materials are also well known in advance, and their behavior under different designs and with different levels of stress assigned can be trivially modeled with CAD packages (and even manually).

Same for weather. It comes down to a few behaviors (rain, wind, earthquakes of various degrees, sunlight) that people can model, and have been modeling for ages. We have 25+ centuries old buildings that still stand.

>How many software projects need teams of construction workers to actually build?

Not sure what you even mean here.

>Different kinds of engineering have different requirements.

That's obvious. The question we ask here is different: sure they are different, but are those requirements equally well defined and equally difficult across software and other engineering fields (like construction)?

And to that I say no. Construction has pretty solid, rarely changing requirements (and almost never changing ONCE construction has begun), and works with specific materials with a limited set of interaction. With software modeling the entire universe is the limit as far as complexity goes.

>How many software projects have to deal with gravity?

A better question is: How many civil engineers need to deal with the fact that gravity keeps changing?

The nice thing about most engineering is that the laws of physics are assumed to be constant.

A program's universe is the hardware it runs on. When that hardware changes, the equivalent of the laws of physics changes.

Amount of RAM, clock speed, number of cores, cache size, etc are all things that you can't assume as constant. Real engineers need not worry too much about deadlocks the way we have to for software. If you have physically moving objects, you can easily design to have things synchronized (gravity doesn't change, friction won't change much, etc).

I have plenty of old software that won't work properly on today's PC's because they can't run on fast computers.

> But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd.

The issue isn't exactly how software fails, but that software today doesn't properly protect confidentiality and integrity of data, ensure services and data are available without providing oracles, authenticate and authorize correctly, log properly, etc. This is a well known methodology that is consistently ignored in preference to shipping products immediately. That's the issue.

Unexpected things happen, accidentally or maliciously, but care should be taken - the proper kind of non-negligent care called due care. In general, software is not fit for the purpose it was created, appearing to work even though it really doesn't.

>> how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that?

Every bit of military hardware, or child's toy, for starters.

More generally, you cite anticipated harm from vandalism or car crashes as being part of the general environment.

It is far past time to recognize that the hacking environment in which we live IS THE NORMAL ENVIRONMENT, and these hacks ARE everyday anticipated harm.

Software does not live in a virtual vacuum of space. It is on the internet, and will likely be scanned withing seconds if unprotected. Even tall buildings are now designed after 9/11 to better withstand jetliners crashing into them. To follow your metaphor, most of the hacks are completely unsophisticated, the equivalent of a bridge falling because some kid came along with a can of spray paint, or a screwdriver and started pulling screws.

Let's also not pretend that other engineering disciplines doesn't also have to face real-world unpredictable and hostile inputs. Even children's toys have to be designed so as not to be hazardous from persistent, random, and clever unexpected attempts at use, misuse, and abuse from children.

The same thing for consumer tools, which have many design features to frustrate the unending supply of clever idiots out there trying to win Darwin awards and sue you if they fail (or their families if they succeed).

I've done both software and hardware, and your expressed attitude that software is somehow inherently more difficult or threatened is utter nonsense. The only basis for this is impatient project management where the key priority is to to ship yesterday because the management are idiots and think software is trivial.

These approaches to start slinging code, write it fast and change it often are useful only for prototyping. In writing real software, my first two steps are 1) understand your problem in depth, and 2) work very hard for a long time to avoid writing code (i.e., better architecture to simplify, separate dependencies, etc.++). After substantial effort is put into those, then write only the code that is necessary. I take a similar approach to manufacturing, understand it, design a process with minimal key elements, then build those.

Software is not a 'special snowflake'.

> It is far past time to recognize that the hacking environment in which we live IS THE NORMAL ENVIRONMENT, and these hacks ARE everyday anticipated harm.

> The only basis for this is impatient project management where the key priority is to to ship yesterday because the management are idiots and think software is trivial.

I agree with the former but it is negated by the latter. The economic incentives certainly do.

> These approaches to start slinging code, write it fast and change it often are useful only for prototyping. In writing real software, my first two steps are 1) understand your problem in depth, and 2) work very hard for a long time to avoid writing code (i.e., better architecture to simplify, separate dependencies, etc.++)

Ideally this would be the case but this approach is a luxury many could not afford. This is especially true in an "Agile", first-to-ship mentality seen in most startups.

I agree that the First-To-Ship mentality you mention and other shortsightedness negate the reality of the current environment.

But if the industry doesn't start changing itself, it'll have change imposed upon it.

The Equifax breach alone, especially if it is weaponized by a state actor has the potential to completely undermine our banking system. All because of weak software and inadequate maintenance.

This is not like we have sophisticated attackers going after solidly designed & engineered structures. It's like we have teenagers and squirrels wandering around the back and freely climbing in windows where the specification didn't even include installing glass. And then commenters like GP saying that it is too great a threat to design around teenagers and squirrels.

Beyond that, the organization they're in sets the priorities; how is it going to fix anything if the engineers are on the hook for Equifax leaks but they still have the same (presumably not very security-conscious) managers?

You raise an interesting question here. Civil engineering these days is done on computers, mostly with incredibly convoluted FEM software and other simulation tools. All software vendors refuse to take any responsibility for bugs in their software and even block you from using the software if you don't agree on the terms.

The problem with software as opposed to most artifacts that humans produce in the world is: "chaos". Sofware is insanely sensitive to initial conditions and a single logical error anywhere along the line leads to... well, chaos[1]. A single errant system may also have insane reach (as opposed to e.g. a bridge), so the consequenses just amplify.

Unfortunately, we've started to rely on the reliability of software in unprecedented ways. (I'd guesstimate that it's probably mostly short-sighted economics that's driving it.)

[1] This is inherent to Turing Machines[2], so if we keep insisting on Turing Complete languages in every realm of computation we'll forever suffer this problem. Please don't think that I'm making an argument against general computing, I'm not. I'm making an argument for restricted DSLs for things that don't need more.

[2] We're not even talking the 'ordinary' chaos of dynamical systems here. Those can at least usually be approximated numerically for the near future unless they're actually really close to a point of divergence. Not so with programs.

> How many engineers live like that?

Well for one thing the engineer planning the electrical isn't the engineer planning the structure. The engineer who scoped the HVAC isn't the guy who wrote the company's website either.

This is a long term problem that we need to begin recognizing and addressing today. I wouldn't expect the programmer who is building the frontend to understand the network security. I wouldn't expect the database guy to know the server administrative role.

Sure there's overlap but to have a team of people whose focus is that specific role is partially what engineers do. As a software engineer myself it's illegal for me to go onto my client's sites and start engineering their electrical system. I pass that effort to my coworkers who are electrical engineers in just the same way they ask me to design them their databases and their network.

> Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that?

I don't think that's a good analogy. Engineers might claim we have it easy because we don't have to face earthquakes or electrical surges or birds flying into our turbines.

Most of the article, and the comment you're replying to, wasn't about hackers, though. It gave the example of people dying because the Therac machine or a Toyota car responded wrongly to non-malicious, normal user input, like a "bridge collapsing unprompted" in normal use.

Yeah, I agree about the article. I wouldn't offer this criticism to the main essay - fundamental bugs in critical systems are absolutely an engineering failure, at least in the sense that we should look for engineering practices to solve them.

But with the post I responded to? I see this idea of "it's because engineers don't care, so give them liability" a lot, and frankly it's stupid. It neglects both cost-benefit differences and any interest in the question of hacking, and then goes on to pretend that you can fix bad systemic pressures by beating up the guys at the bottom. The difficulty of writing good code is not a function of how heavily you punish bad code.

I flatly don't believe that the Therac bug would have been caught by personal liability for the bug; either the code would have been cancelled altogether to avoid liability or the bug would have gone on unabated. Realistically, criminal liability for lethal bugs mostly sounds like a way to drive talented devs away from high-stakes applications and leave those apps to be handled by people desperate enough to run the risk, or by international shops operating outside the reach of prosecution. I don't love the thought of a shop unwilling to use git because it creates personal responsibility. "Oh, we just pass code around on thumbdrives, I can't remember who wrote that critical line."

I'd shrug and let the whole thing go, but the rumblings of "regulate software" keep getting stronger from a lot of directions. I don't really want to wake up in a decade and find out that degree requirements and expensive-but-useless licensing have become mandatory, and it feels like pretending bugs come from lazy devs is another snowflake in that avalanche.

>The only difference is that these other engineers are motivated to be significantly more thorough about their jobs.

Well, it's a whole ecosystem problem. Most of the places I've worked, security was just supposed to be something you learned and credentialized on your own, and no time or resources were deliberately allocated to it.

In fact, you're likely seen as an underperformer if you slow down to get things right or to refactor bad code. And when software does break, `git blame` tends to accuse the developer that was in the worst position to fix a ship that had begun sinking long ago.

I'm not sure how engineer culpability works in other fields, but surely it's always coincided by bad process and management at which point the whole company is responsible.

In professional engineering the accountability comes with a stamp, and a professional ethics requirement to report up the chain if a critical problem has been found that could impact public safety. This includes whistleblowing if management ignores the problem.

There are clear guidelines as to what types of products and services require a p.eng.'s stamp. There are also many professional engineers out there who's stamp has never touched ink. For example, the engineers drafting and churning out spreadsheets in the engineering sweatshops aren't accountable for the safety of the product, you'll have one or more engineers providing their stamp for that group's work.

If we were to implement a similar model on software development, most developers would not have personal liability for work they do e.g. for a company or open source project. What is needed is better classification of software products/activities that require a software engineer's stamp[1], and a corresponding professional association.

So no, developers should not be accountable. However, certain safety-critical products - where safety can be defined in various ways expanding the current relatively small set of NASA, aviation and health-care software - need engineers overseeing design, development, and QC to be accountable.

And yes, this means that the self-taught developer, no matter how talented, will not be able to release certain types of software without some sort of oversight.

[1]As a side-note - not everyone can call themselves an 'engineer' in other fields without the proper credentials and professional membership. I see developers calling themselves software engineers all the time though they don't have the chops. This would, of course, need to change.

Updated to change asterisk to brackets

Regulation on programmers is not the solution.

Regulation on companies might be. I'm usually really disappointed when a cheap re-seller takes a government purchase of a 911 or 112 ( here in EU ) and then hires a subcontractor, which hires a subcontractor, which ...

That's how crappy software gets into production.

Without radical, NASA-style changes to typical development, where each line spends much more time being audited than being written, it is completely unreasonable to expect that kind of liability from developers.

Ditch the dumb deadlines and the tight budgets and it would become a hell of a lot more feasible.

I WISH I had the time and budget to put due diligence into my work.

You have to think about what you're building too. Feature release would come much slower than it does today. Reasonable? If you're writing software that is absolutely safety-critical, that's likely. If you're writing iTunes, well, I have my doubts.

It's not like reliable, secure software doesn't exist. It's just limited to spaces where people are willing to use years-old tech, forgo new features, and pay fortunes for auditing. Air travel, industrial machinery, medicine, and so on.

Almost like the cost-benefit actually doesn't support making the development of trivial software ten times slower and more expensive...

If there's a problem here, it's business externalities (Equifax won't pay adequately for screwing people), not engineering impossibility (the rest of us know how to patch Apache).

I mostly agree, but this article cites the national 911 software, which arguably should be treated as a critical system, failing, and some medical devices whose software killed people.

Yeah, fair enough - I've mostly been responding to too-general comments, the article is more specific.

The 911 thing seems like an obvious issue that absolutely should have been caught by a rigorous development system; it was treated like a digital phone system instead of a life-saving service. The Therac bug sort of feels like a different story, because that bug led to IEC 62304 - it's older than the practices which would now prevent it. They're both critical systems that were written without due care, though.

Broken software does not typically kill people or makes you loose money (time or some other resources, maybe).

In contrast bridges that fall apart tend to kill people in a nasty way. They are also quite hard to reboot.

An excellent framing I saw recently was "I can tell the people who write these articles don't actually want secure code, because they aren't paying me to write it."

There's plenty of carefully-written, rigorously-tested code out there. Want to write software for air travel, drug manufacturing, or space flight? You'll have no choice but to write safe, first-rate code.

Want a smartphone weather app written to those standards? Well, for $50 and a feature set five years behind the state of the art, you can have it.

Isn't this a tooling and knowledge issue? If I was as productive with TLA+ or model based design as I am with React-Native/Ionic/Swift etc. then we could have a smartphone weather app written to those standards for the same price.

While there's plenty of room for improvement, I legitimately believe that TLA+ is now in a place where it's convenient enough to use in conjunction with Ionic/Swift.

Broken software has the capability to regularly kill people or cause people to lose money. It happens too often.

I'm pretty sure less people would be rushing into the software engineer profession if they knew it came with the liabilities you described. That would drive up the prices of engineers.. and I'm pretty sure the industry in general doesn't want that.

Maybe that's why the free pass mentality exists

Or maybe the "free pass mentality" exists because development doesn't happen in a vacuum and errors don't necessarily find their way into programs because the programmers are stupid or careless.

Can't it be both a technology problem and an accountability one? The tech problem might take ages to get solved without accountability forcing it. If we were accountable, we'd be forced to make better tools for writing reliable software more easily. That said, there are simpler low handing fruit "tools" like code review, safe git practices, etc that still aren't followed everywhere. Ignoring those low hanging fruit does point to accountability/prioritization being a cause of low reliability.

Motivation seems far from the only difference.

If we divide making into "builds with atoms" and "builds with bits", both sorts of things are generally unregulated. It's when you get into "builds life-critical things with atoms" that you see significant regulation. If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't.

I agree that societal accountability for engineers working on life-critical system, software and hardware both, is vital. For most atom-making and bit-making, though, I think stringent professional licensing is overkill. Instead I'd rather we started with accountability for the people who have much more control over the outcomes than individual software engineers: executives and managers.

> If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't.

A filing cabinet must be fit for the purpose it is used, or there will be liability incurred. OSHA would frown on keeping hazardous chemicals in your filing cabinet. Software should also be fit for the purpose it is marketed and used.

There are many kinds of paper, but paper being used improperly, by storing a list of credit card numbers on it, is quite similar to how many pieces of software are written.

You can definitely do that. You just have to pay them 20x more money and give them 50x more time and give them absolutely all the tools they need. Best possible hardware. Best possible software. Fastest possible internet connections. etc.

Show me an employer willing to do that and I will immediately sign up.

Most companies want to do the exact opposite. Spend the least amount possible, because, you know, the budget is tight and the business side wants all these features ready yesterday, nay, last week.

>The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation.

Regulation like that tends to follow after a series of disasters causing massive loss of life. Software hasn't yet managed to achieve that.

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs.

What evidence supports this as the only difference? I would argue these other engineers have significantly smaller jobs to do, so it's easier to be thorough.

Edit: Why are you so keen to fix a problem you don't understand? Are you a programmer?

You can't drop all of this responsibility on developers. You need people checking behind them.

But if we want to pay developers like surgeons and have malpractice insurance, all the better, I guess.

Reply on news.ycombinator.com