Uber Self-Driving Car That Struck Pedestrian Wasn’t Set to Stop in an Emergency

I've seen a few people comment on the footage that they too would have run the pedestrian over, to which my only response is: I sure hope you don't have a driver's license [anymore]!

This doesn't mean that Uber therefore did the right thing in disabling the system; it probably means that the system shouldn't have been given control of the car in the first place. But my point is that there is no readiness level where driverless cars will ever be safe -- not in the same way that trains and planes are safe. The driving domain itself is intrinsically dangerous, and changing the vehicle control system doesn't change the nature of that domain. So if we actually care about safety, then we need to be changing the way that streets are designed and the rules by which they are used.

And that is why I am so mad at Uber. They are compromising the public trust in autonomous cars with their reckless release policy. And thereby potentially endangering even more lives, as we have to convince the public of the advantages of this technology.

It hasn't shown that at all. It has documented beyond reasonable doubt that Uber should not be allowed to participate in real world tests of autonomous vehicles.

There are plenty of situations where people would fully accept a self driving vehicle killing someone but this isn't one of those.

Uber had a fatality after 3 million miles of driving.

The mean fatality rate is approximately 1 per 100 million miles of driving.

It's a sample size of one, so the error bars are big, but it drives me insane that people are acting like the Uber cars are the ideal driverless cars of the imagined future, and are super safe. The available data (which is limited, but not that limited) is that Uber driverless cars are much, much, much more dangerous than mean human drivers.

That actually sounds like a really interesting concept, one of those ideas that seems obvious only after someone suggests it. What company is this?

Right now, in the Seattle area, we are basically seeing a new littering epidemic in the form of sharable bicycles being left to rust away, unused, at random places. If the bike could cruise to its next user autonomously, that would be really be a game-changer. "Bikes on demand" would turn bikesharing from (IMHO) a stupid idea into something that just might work.

Plus, the engineering challenges involved in automating a riderless bicycle sound fun.

What the Uber crash has shown us is mostly the willingness of people on HN to excuse Silicon Valley darlings even when they actually demonstrably kill people.

This is an analogy that cannot completely map to cycling.

A fall at any speed from a bike is literally a potentially crippling or deadly scenario.

That depends, there could simply be no traffic behind you, which an experienced driver and hopefully and automated one would be monitoring.

Besides, there are many situations on the highway where an E-stop is far safer than any of the alternatives even if there is traffic behind you. Driving as though nothing has changed in the presence of an E-stop worthy situation is definitely not the right decision.

And "almost" is always a good idea when talking about a future that looks certain. Takes into account the unknown unknowns. And the known unknowns (cough hacking cough).

If you want to compare it with a car operating on cruise control you'd have to sedate the driver.

Or at least it should be. That's why there is a v_max that a car is not allowed to exceed and a faster or heavier car will have better breaks. And 70 km/h as here should be far away from v_max.

The way I see it,there is only way to make sense of a field where the most respectable R&D house (Google/Alphabet) limits their vehicle to a relative snail's pace while everyone else (including notoriously unethical shops like Uber) is taking a gung-ho, "the only limit is the speed limit" approach. That is to assume "everyone else" is cheating the game by choosing a development path that gives the appearance of being functional in the "rough but will get there with enough polish" sense while the truth is that it's merely a mountain of cheap and dirty hacks that will never achieve the goals demanded by investors.

The only reason a company would overlook such a simple safety protocol as "slow down until a potentially dangerous object is positively identified" is if their "AI" fails to positively identify objects so frequently that the car could never achieve what a human passenger would consider a consistent, normal driving pace. The same can be said for any "AI" that can't be trusted to initiate panic braking in response to a positively identified collision scenario with a positively identified object. The fact that they specifically wrote an "AI" absolving workaround for that scenario their software means the frequency of false positives must be so high as to make the frequency of "false alarm" panic braking incidents unacceptable for human passengers.

"As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path."

1. It seems like the classifier flipped state between pedestrian/unknown object/vehicle/bicycle; this seems like one of the well-known issues with machine learning. (I'm assuming the classifier is using ML simply because I have never heard of any other (semi-?) successful work on that problem.)

I suggest that the problem is that the rest of the driving system went from 100% certainty of A to 100% certainty of B, etc., with a resulting complete recalculation of what do to about the current classification. I make this hypothesis on the basis of the 4+ seconds when the car did nothing, while a response to any of the individual possibilities would possibly have averted the accident.

2. If the classifier was flipping state, I assume the system interrupted the Decide-Act phases of an OODA loop, resulting in the car continuing its given path rather than executing any actions. This seems like a reasonable thing to do, if the system contains no moment-to-moment state. Which would be strange; it seems like the planning system should have some case for having obstacles A, B, C, and D rapidly and successively appearing in the same area of its path.

3. Assuming the classifier wasn't flipping state, but presenting multiple options with probabilities, I can see no reason why the car wouldn't have taken some action in the 4+ seconds. (I note that the trajectory of the vehicle seems to move towards the right of its lane, which is a rather inadequate response and likely the wrong thing to do for several of the classification options.)

"According to Uber, emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior."

That's just idiotic and would be nigh-criminally unprofessional in most engineering situations.

What makes you think they aren't sent to VW all the time?

This is also an issue with current production automatic braking system. One that was largely solved with on track testing, and on road testing logging false triggers with a driver driving. There's no need to risk lives unless you're just cutting corners to avoid the cost of a test track.

I'm not sure that'd be a huge issue. The vectors have to be intersecting first of all, which most vectors emanating from sidewalks wouldn't be, and then a little hysteresis would smooth out most of the rest.

"It might move out of my way" is no reason to get so close at such a high speed that you can't avoid it when it doesn't!

https://www.nbcnews.com/tech/innovation/emails-show-arizona-...

The police are in no position at assert that, nor do they know whether or not Uber is guilty of negligence. Police do not bring charges and they're not running the investigation.

The place for the product folks to override safety features is the test track. If the feature didn’t work, they should have pulled the drivers because they were not trained to properly operate the machine.

If you give the “driver” training on a car with an autonomous braking system, then give them a car without it, that’s not on the driver. Someone was negligent with safety in regards to the entire program.

I’m not saying anyone needs to go to jail over this, but there do need to be charges IMO. Personal liability needs to be involved in this or executives will continue to pressure employees to do dangerous things.

That should be irrelevant. Even if the pedestrian is jay-walking, it's still not legal to hit them. Further, having solid evidence that the car detected the pedestrian and did nothing to avoid her mitigates the pedestrian's responsibility, no?

Also, the "center median containing trees, shrubs, and brick landscaping in the shape of an X" sure looks like it should have some crosswalks, from the aerial photograph. What's it look like from the ground?

And partially-at-fault, on one hand, means there's fault on the driver side too, and on the other hand, is a judge's decision to make, not the police, no?

That's like creating HTML forms that only work with the common use case and crash spectacularly on unexpected input. Except that this time it's fatal. That's not the kind of software quality I want to see on roads.

This is beyond incompetence. There is a different level of software engineering when making a website vs making a pacemaker, rocket or flight avionics. You need the quality control of NASA, SpaceX or Boeing, not that of .. whoever they have running their self driving division.

Yeah that's not how you design safety critical software. This isn't some web service. Either you're wrong (let's hope) or Uber is completely negligent.

Source: I write safety critical code for a living.

In no case would sudden braking be the cause of a rear-end collision. It's always the fault of the driver behind.

The regulators should only test for false negatives, where the car should have stopped, but did not detect the obstacle (false negative), because there, it is a clear threat to safety, and the car company's incentive, while definitely still present, is less pure, as the amount of false negatives is a direct trade off with the quantity of false positives (because it is a treshold: a minimum confidence level from which you decide that there is indeed something in front of the car and you need to break), which make driving more awkward for 99% of drivers

1. Regulators want aggressive breaking but carmakers want smoother driving

2. Manually tuning all the edge cases for cases where the software is uncertain what's happening will lead to fragile monolith black boxes

The charitable interpretation of this is that the industry believes that self-driving AI is a safety feature of greater quality than, say, lane assist or auto-breaking.

The less charitable one is that they find it too much work to integrate their AI to other safety systems. Which, to be fair, is really going to be a lot of extra work, on top of developping self-driving.

"emergency braking maneuvers" refers to an additional automated (software) system for automatically applying the brakes in an emergency (that's detected by that additional system).

If any of the systems are vulnerable to any such false positives, and equipped to enable emergency braking to avoid them, even on the highway, it's not hard to imagine why they might be disabled, especially during testing phases.

I think it's fair to say that at this early stage in product development, there's probably no 'always right' answer for how to handle a given obstacle without considering locality, driving speed, road conditions, likelihood of false positives and negatives, etc.

Instructions from TCAS are near absolute in their priority. If ATC says to do something different, you ignore ATC and do what TCAS says. If the pilot in command says to do something different, you ignore the pilot in command and do what TCAS says. If somehow God Himself is on your plane and tells you to do something different, you ignore Him and do what TCAS says. Compliance with TCAS is non-negotiable, and the Überlingen disaster[1] is the bloody example of why it's that way.

Self-driving/autonomous-car systems need to have a similar absolute authority built in. If Uber disabled theirs because of false positives, it's a sign Uber shouldn't be running those cars on public roads.

[1] https://en.wikipedia.org/wiki/2002_%C3%9Cberlingen_mid-air_c...

Redundant safety systems are a great idea, evidently Uber needed more of them, and integrating with the Volvo system might have been a reasonable option. It's silly to suggest that the integration would necessarily have been trivial, though. That's what I'm objecting to.

The point is that so far most (all?) of these computer systems don't have a failure mode where it kills pedestrians, because their scope is very limited.

I think the better interpretation is that the Uber system disabled another separate (non-Uber) system.

I didn't say you can't drive a car without being cautious.

I said you can't drive it without constantly braking the moment you detect an object, irrespective of what the object is doing. (e.g. moving into or away from the driving path).

i.e., just because an object was detected 6 seconds before impact did not mean the car ought to have started braking at that moment. It could be that the object was 200 feet away and moving away from the car's driving path, 6 seconds before impact. It'd be absolutely ridiculous to brake in that situation.

We have no information about this context, e.g. the car's data or determinations within the 6 second window. We only know it detected an object 6 seconds before impact.

It appears like the person I was replying to implied 'the braking distance was 180 feet, but the person was 380 feet away, thus uber could have prevented killing this woman had it not shut off the brakes'. In reality, the 6 second figure isn't relevant. What is relevant is the context that allowed a reasonable driver/AI to determine at a particular point in time, that the car should have slowed/braked. And we don't have that information yet. That's what I'm interested in.

The point I was trying to make is that I believe the noteworthy event is that a self-driving car struck a person, without even attempting to avoid it. That she was killed is a tragic twist to the story. I believe that even if she wasn't killed it would still be a noteworthy event because, again, the car did not attempt to avoid it.

I do not at all intend to diminish the fact that Uber killed someone.