Hacker News Reader

We can write safety-critical software. Look at NASA’s CMM level 5 processes for some of their flight control software.

I’m not as familiar with applications in other domains, but I know they exist. This is typically not fun code to write. It is bureaucratic and developed over decades. It is more an engineering discipline than most commercial software. And, it is organizational, this is not code written by a single individual. I would not trust that at all. More eyes mean more trust, broadly speaking.

The 737 Max issues were not as much about software as corporate cost saving efforts.

I have no idea if the OneWheel software can be fixed. I’m sure there was real talent that went into it. I suspect the first to market and other financial incentives were big factors in this outcome. Just the fact they are only now, apparently, adding user notification of error states is kind of crazy.

So I agree that certain types of organizations are not trustworthy, but I also think the safest code we have ever written was done by…organizations.

NASA’s code runs on devices that have am very specific missions. They can test all of the edge cases because they’re going from A to B once and they have a decade to prepare that journey. People riding electric scooters and etc will run into more variability each day than NASA will in a whole mission.

I crashed and broke my back on a two wheeled scooter in SF due to a pedestrian walking on a red don’t walk sign, for example.

Plus, none of these companies are doing NASA quality assurance or anything close, so why even compare them to NASA?

I’m saying that organizations can, in fact, write safety critical code. One wheel does not need to account specifically for pedestrians crossing the road, but it does need to behave properly under a wider variety of conditions than a spacecraft or launch vehicle. Give it a decade with a half dozen teams working to making the code correct, and I think you could get there.

I am inclined to think that a commercial entity cannot reliably write safety critical code for a primary income driver. You need different financial incentives and probably a different organizational structure for this to work.

At the risk of derailing my point in minutiae, I suspect Waymo will ultimately deliver safe self-driving technology sooner than Tesla, for example.

> I crashed and broke my back on a two wheeled scooter in SF due to a pedestrian walking on a red don’t walk sign, for example.

And how exactly it was software's fault ?

It's not, it's an illustration of the variability faced riding "electric scooters etc." versus controlling a space shuttle:

> People riding electric scooters and etc will run into more variability each day than NASA will in a whole mission.

> I crashed and broke my back on a two wheeled scooter in SF due to a pedestrian walking on a red don’t walk sign, for example.

Yes, the point is that even without bad software, the world is a dangerous place. Better to not have unsafe, unstable, dangerous vehicles to compound that. It was part my fault and part the pedestrians fault that I crashed, but the real cause was the instability of the platform. If I had been on a bike, I’m certain I could have stopped or avoided the pedestrian.

The thing is this code only has to do one very simple function. It should be a few hundred lines at most. It's easily within the realm of formal verification.

I guess they always thought they didn't need it...

Getting the code right to compute how to drive the outputs when you have good inputs isn't the hard part. It's what happens when things overheat or time out or disconnect or get noisy that's hard; usually you'd just reset but the Onewheel can't do that while you're driving.

Yes that's an obvious fundamental limitation of the entire design - it isn't remotely fail safe. You can't really fix that. You can reduce the chances of failure with redundant hardware and formal verification of the software.

It's not at all clear what the actual problem is here but in this thread we were discussing writing bug free software.

No, formal verification knows nothing about the hardware, which is required for the system to function correctly. This is "not even wrong"

No it is not. You are mistaken. The comment I was replying to was talking about software bugs.

It might be the case that the problem is a hardware issue but the article is extremely unclear about what the problem really is (maybe nobody knows?) and they claim to have fixed it with a software patch.

I'm familiar with similar can't-fail applications in industrial controls engineering. Safety-critical software usually only means enough watchdogs, counters, and checksums that you can be confident everything will shut down if something's not right.

But the Onewheel is a harder problem - you must continue to run even if something goes wrong! There are some industrial processes that are run continuous, you must not ever stop that pump or furnace or the molten metal will solidify or sewage will back up into people's homes or things will otherwise go very, very wrong. To deal with that, you need at the very least one-out-of-two function, redundant heterogeneous parallel motor drivers and IMUs in the Onewheel for example. Most plants - even big automotive lines, where an isolated fault can cost tens of thousands of dollars per hour - do not go to that enormous effort and expense. A consumer product never will.

I feel like this is an engineering cultural issue. Few organizations can produce safety-critical code, say organizations who focus on tail events and long term profitability. And an engineering organization cannot transition from a growth oriented shipit culture to a safety-critical culture.

Just to clarify, they are not just now adding feedback about reaching the limits of the board. Since day one they have had “pushback” which tilts the board backwards in a very noticeable way to let the rider know that a limit (speed, power, torque, battery, temperature) is about to be reached. Unfortunately many, maybe even most riders intentionally ignore this warning and forcibly “push through” the pushback. This update just makes the warning much more annoying in an attempt to change rider behavior.

> This reminds me of those viral videos of indoor cyclists that were going around >10 years ago

The other day my wife showed me this unbelievable performance of a Chinese acrobat unicycling on top of a big rolling globe while doing acrobatics with balancing bowls on her head [1]. Since that site is Chinese language with weird popups, I also found a 10 years later performance by the same lady on youtube [2], but it's not quite as effortless as the first.

[1] https://www.douyin.com/video/7281815910430657846

[2] https://www.youtube.com/watch?v=FYuRlFjwql4

>that site is Chinese language with weird popups

That site is tiktok or rather the Chinese version of it.

I saw a dad with his son together on one (or a similar product) last week. I doubt that they test things like that thoroughly.

It definitely looked like an accident waiting to happen. Both were wearing helmets but they were going pretty fast. I can imagine that smashing your face into the ground will be ugly at those speeds. Especially for a 6-7 year-old with the weight of an adult added on top...

Your comment reminds me of the Boeing 737 MAX problems which resulted in them all being grounded worldwide.

Reminded me of this too. But luckily those 737 max8 got a software update (and a new name) and are happily flying again...

I flew in one a few months ago and was really pissed off that I didn’t know about it ahead of time. I thought when they announced the plane we were on that it was a bad joke (flight had already been delayed many hours). Luckily nothing bad happened but I had to quickly google how they resolved the issue.

It got more than that, some more redundant sensors, pilot training specific to the plane, and an option to override the system.

They always had the option to over-ride the system: turn off the electric trim, same as for runaway trim (which the failure much resembled).

That was not a software bug, but a bad design from a dysfunctional organization that had long since stopped valuing sound engineering.

And they even had the warning and procedure for the rare situation if it happened!

Onewheel is just going to apparently warn you “this thing may not work. Whoopsie!”

The "skin in the game" point is interesting. Imagine if there was a way to buy personal injury insurance for a device such as the Onewheel. Then the insurer has an incentive to audit the code and check for safety hazards, in order to figure out what premium to charge. If you're a customer and you want to know which device is the safest, check to see which has the cheapest insurance.

Seems it would be easier and more effective to handle this at the source.

Set some minimum standards for the products and require a bond or insurance for them to be sold.

We already set standards for other vehicles and safety critical products (FAA, FMVSS, UL listing, etc). The minimum standards makes a case for culpability through negligence relatively easy.

Requiring a bond or insurance means either betting a substantial amount of your own resources on the fact that you have done correctly, or convincing an insurance company (who are demonstrably passable at assessing risk based on their still being in business) to assess the likelihood of payouts and price your premiums accordingly. If you can’t convince them you made this safe, you won’t be able to afford to put it on the market.

Then instead of expecting somehow that OneWheel is going to cooperate with audits they have no real incentive to participate in, consumers in aggregate to make informed, rational decisions, and a bunch of other things that are unlikely to happen… (The insurance company is just going to base rates on aggregate historical data like they do with cars. Some of you may need to die so we can determine whether the products are safe. Best of luck.) People can just go buy products at the store and be relatively confident that they’re not going to kill them.

I’d wager that markets are generally healthier when people have confidence that generally available products aren’t dangerous. I can only imagine how little I’d purchase if instead of electrical standards and UL listing I had to go talk to my home insurer every time I wanted to plug something else into the wall so I could find out if it was going to kill my family.

Why do people get in cabs with crazy cab drivers? There's even a game called "Crazy Taxi"

Because they are confident that the cab driver doesn't want to kill HIMSELF. If he behaves recklessly, he'll face consequences

That is, self-driving cars with remote operators are a moral hazard. If they make a mistake, the operator doesn't face any consequences

This is not a trivial point. If you look deeply into causation, you'll see it in many places

---

Likewise, Tesla's policies of forcing your hands on the wheel are incoherent. They claim to drive you around, except that it's your fault if something goes wrong. Well, that's exactly where I wanted the help.

Definitely check out "Skin in the Game" by Taleb if you haven't

I predict that this will become a recognized limitation of AI. In 10 or 20 years, we'll realize that it was a philosophically incoherent position to claim that a machine is doing something, WITHOUT being liable for the consequences of performing poorly.

And not just incoherent, but bad engineering. Without this vital feedback, it won't operate well, and it won't learn. In evolution, which is the process that has produced machines capable of navigating the world (us), death is a very strong piece of feedback!

The problem is that it takes time. Learning how to go on 2 wheels isn't that hard

Learning how to go on one is noticeably harder as in need 2 axis of balancing, not (essentially) one

Meanwhile they want to sell product that's small and convenient enough to carry, and one wheel gives it advantage in that. And artificial balancing is easiest way to add it.

> I mean think about SLOW the self-driving cars are after 10+ years. Their reaction time and judgement is shit.

> I'm not an exceptional cyclist, but I've learned to make decisions safely in 15+ years of riding around SF, through diverse conditions and terrain. It's obvious to me that these capabilities are beyond software.

This thing is not self-driving. There is MASSIVE gulf between "make a vehicle controllable by human" and "make that vehicle self drive".

All true, but you have to trust the organization(s) that manufactured your bike. And unless you’re paying $Ks of dollars the manufacturing quality is sadly going down.

My main point is: You gotta trust someone. If you trust no one you should move out to a remote uninhabited forest.

No, it's not all or nothing. Good engineering is simple, relies on less, and trusts less -- it doesn't require trusting zero.

I also addressed the difference in the comment -- this type of software has no end-user / operator repair, and the costs of failure are great.

I know how a bike works, and I can repair it. (Time-tested materials like steel frames are also nice.)

I don't know how the OneWheel works. In theory, I might be able to read some of the source code, but that counts for very little. It's certainly more complex than a bike. It also hasn't been tested over a long period of time.

Well it looks like we've had a test... I haven't touched a OneWheel since I first saw them 5 years ago, and now it seems like I have some good info about them.

The test of time works!

>And unless you’re paying $Ks of dollars the manufacturing quality is sadly going down.

Nope, especially if you’re paying $Ks of dollars the manufacturing quality is sadly going down.

Totally not, I can take a bicycle into my workshop, check if is made of steel or aluminium, inspect quality of hardware, I can test quality of a weld, hit it with a hammer, etc..

You can't see source code of onewheel, or take out the firmware and run it in isolation.

Software producing companies have the privileges of being protected from inspection, but responsibility of a guy assembling Ikea furniture.