Comment by flatline | Hacker News Reader

Comment by flatline | Hacker News Reader

Parent Full thread

We can write safety-critical software. Look at NASA’s CMM level 5 processes for some of their flight control software.

I’m not as familiar with applications in other domains, but I know they exist. This is typically not fun code to write. It is bureaucratic and developed over decades. It is more an engineering discipline than most commercial software. And, it is organizational, this is not code written by a single individual. I would not trust that at all. More eyes mean more trust, broadly speaking.

The 737 Max issues were not as much about software as corporate cost saving efforts.

I have no idea if the OneWheel software can be fixed. I’m sure there was real talent that went into it. I suspect the first to market and other financial incentives were big factors in this outcome. Just the fact they are only now, apparently, adding user notification of error states is kind of crazy.

So I agree that certain types of organizations are not trustworthy, but I also think the safest code we have ever written was done by…organizations.

NASA’s code runs on devices that have am very specific missions. They can test all of the edge cases because they’re going from A to B once and they have a decade to prepare that journey. People riding electric scooters and etc will run into more variability each day than NASA will in a whole mission.

I crashed and broke my back on a two wheeled scooter in SF due to a pedestrian walking on a red don’t walk sign, for example.

Plus, none of these companies are doing NASA quality assurance or anything close, so why even compare them to NASA?

I’m saying that organizations can, in fact, write safety critical code. One wheel does not need to account specifically for pedestrians crossing the road, but it does need to behave properly under a wider variety of conditions than a spacecraft or launch vehicle. Give it a decade with a half dozen teams working to making the code correct, and I think you could get there.

I am inclined to think that a commercial entity cannot reliably write safety critical code for a primary income driver. You need different financial incentives and probably a different organizational structure for this to work.

At the risk of derailing my point in minutiae, I suspect Waymo will ultimately deliver safe self-driving technology sooner than Tesla, for example.

> I crashed and broke my back on a two wheeled scooter in SF due to a pedestrian walking on a red don’t walk sign, for example.

And how exactly it was software's fault ?

It's not, it's an illustration of the variability faced riding "electric scooters etc." versus controlling a space shuttle:

> People riding electric scooters and etc will run into more variability each day than NASA will in a whole mission.

> I crashed and broke my back on a two wheeled scooter in SF due to a pedestrian walking on a red don’t walk sign, for example.

Yes, the point is that even without bad software, the world is a dangerous place. Better to not have unsafe, unstable, dangerous vehicles to compound that. It was part my fault and part the pedestrians fault that I crashed, but the real cause was the instability of the platform. If I had been on a bike, I’m certain I could have stopped or avoided the pedestrian.

I'm familiar with similar can't-fail applications in industrial controls engineering. Safety-critical software usually only means enough watchdogs, counters, and checksums that you can be confident everything will shut down if something's not right.

But the Onewheel is a harder problem - you must continue to run even if something goes wrong! There are some industrial processes that are run continuous, you must not ever stop that pump or furnace or the molten metal will solidify or sewage will back up into people's homes or things will otherwise go very, very wrong. To deal with that, you need at the very least one-out-of-two function, redundant heterogeneous parallel motor drivers and IMUs in the Onewheel for example. Most plants - even big automotive lines, where an isolated fault can cost tens of thousands of dollars per hour - do not go to that enormous effort and expense. A consumer product never will.

Just to clarify, they are not just now adding feedback about reaching the limits of the board. Since day one they have had “pushback” which tilts the board backwards in a very noticeable way to let the rider know that a limit (speed, power, torque, battery, temperature) is about to be reached. Unfortunately many, maybe even most riders intentionally ignore this warning and forcibly “push through” the pushback. This update just makes the warning much more annoying in an attempt to change rider behavior.

I feel like this is an engineering cultural issue. Few organizations can produce safety-critical code, say organizations who focus on tail events and long term profitability. And an engineering organization cannot transition from a growth oriented shipit culture to a safety-critical culture.

The thing is this code only has to do one very simple function. It should be a few hundred lines at most. It's easily within the realm of formal verification.

I guess they always thought they didn't need it...

Getting the code right to compute how to drive the outputs when you have good inputs isn't the hard part. It's what happens when things overheat or time out or disconnect or get noisy that's hard; usually you'd just reset but the Onewheel can't do that while you're driving.

Yes that's an obvious fundamental limitation of the entire design - it isn't remotely fail safe. You can't really fix that. You can reduce the chances of failure with redundant hardware and formal verification of the software.

It's not at all clear what the actual problem is here but in this thread we were discussing writing bug free software.

No, formal verification knows nothing about the hardware, which is required for the system to function correctly. This is "not even wrong"

No it is not. You are mistaken. The comment I was replying to was talking about software bugs.

It might be the case that the problem is a hardware issue but the article is extremely unclear about what the problem really is (maybe nobody knows?) and they claim to have fixed it with a software patch.

Reply on news.ycombinator.com