Self-driving Uber car kills Arizona woman crossing street | Hacker News Reader

Self-driving Uber car kills Arizona woman crossing street

reuters.com

2,361 pointskgwgk1,766 comments

One aspect that comes from this is that now car crashes can be treated more like aircraft crashes. Each self-driving car now has a black box in it with a ton of telemetary.

So it's not just "don't drink and drive", knowing that they'll probably reoffend soon anyway. Every crash and especially fatality can be thoroughly investigated and should be prevented from ever happening again.

Hopefully there's enough data in the investigation so that Tesla / Waymo and all other car companies can include the circumstances of the failure in their tests.

Every lesson from aviation is earned in blood. This death wasn't necessary though. The Otto/Uber guys have been informed about their car's difficulty sensing and stopping for pedestrians. I know this because I informed them myself when one almost ran me down in a crosswalk in SF. Can't learn anything from your lessons unless you listen. Maybe they can pause and figure out how to actually listen to their reports of unsafe vehicle behavior.

Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car... there is no mechanism to share that learning with the other car companies. There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures.

Also my problem with this is that a human death is functionally treated as finding edge cases that are missing a unit test, and progressing the testing rate of the code... and that really bothers me somehow. We need to avoid treating deaths as progress in the pursuit of better things

I hope we see the day when every car crash makes national news and there’s a big NTSB investigation into what happened.

NTSB are investigating this incident including electronic recorders https://www.ntsb.gov/investigations/Pages/HWY18FH010.aspx and they have another investigation from 2017 still to report https://www.ntsb.gov/investigations/Pages/HWY18FH001.aspx

With Uber's reputation, I wouldn't be surprised if they try to write an app to falsify black box telemetry in the event of a crash to put the liability on the victim. Maybe they'll call it "grey box" or "black mirror".

Does the NTSB have regulation on how black boxes are allowed to function?

> "Every crash and especially fatality can be thoroughly investigated"

Would be better if the code and data was open for public review.

https://news.ycombinator.com/item?id=16547215

This comment by Animats in the context of self-driving trucks is quite telling. He warns precisely of this danger.

I'm hijacking my own post, but this is a very relevant MIT lecture on 'Technology, Policy and Vehicle Safety in the Age of AI' [0].

[0]: https://www.youtube.com/watch?v=LDprUza7yT4

> and should be prevented from ever happening again.

Take a look at the cost of airplanes, even small ones.

Black box with a ton of telemetry being piped into black box models though.

To ensure that all automotive software incorporates lessons learned from such fatalities, it would be beneficial to develop a common data set of (mostly synthetic) data replicating accident and 'near miss' scenarios.

As we understand more about the risks associated with autonomous driving, we should expand and enrich this data-set, and to ensure public safety, testing against such a dataset should be part of NHTSA / Euro NCAP testing.

I.e. NHTSA and Euro NCAP should start getting into the business of software testing.

Dr. Mary Cummings has been working on talking to NHTSA about implementing V and V for autopilots/AI in unmanned vehicles for a few years now. She's also been compiling a dataset exactly like what you are talking about.

I think the idea is to build a "Traincar of Reasonability" to test future autonomous vehicles with.

You might want to check out her research https://hal.pratt.duke.edu/research

Why would Uber agree to such regulations?

They were unwilling to legally obtain a self-driving license in California because they did not want to report "disengagements" (situations in which a human driver has to intervene).

Uber would just set their self-driving cars free and eat whatever fine/punishment comes with it.

There may eventually be standard test suites that can be applied to any of the self-driving systems in simulation. This would give us a basis of comparison for safety, but also for speed and efficiency.

As well as some core set of tests that define minimum competence, these tests could include sensor failure, equipment failure (tire blowout, the gas pedal gets stuck, the brakes stop working) and unexpected environmental changes (ice on the road, a swerving bus).

Manufacturers could even let the public develop and run their own test cases.

How about not testing it on the unpaid public in incremental patches like this age of software "engineering" has decided it was a good idea to do?

This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one.

So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails.

Moreover, the human brain won't like processing these freak accidents. People die in car crashes every damn day. But we have become really accustomed to rationalizing that: "they were struck by a drunk driver", "they were texting", "they didn't see the red light", etc. These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention".

But these algorithms will not fail like that. Each accident will be unique and weird and scary. I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road. It'll always be tragic, unpredictable and one-off.

Very little of what goes into a current-generation self-driving car is based on machine learning [1]. The reason is exactly your point -- algorithmic approaches to self-driving are much safer and more predictable than a machine learning algorithms.

Instead, LIDAR should exactly identify potential obstacles to the self-driving car on the road. The extent to which machine learning is used is to classify whether each obstacle is a pedestrian, bicyclist, another car, or something else. By doing so, the self-driving car can improve its ability to plan, e.g., if it predicts that an obstacle is a pedestrian, it can plan for the event that the pedestrian is considering crossing the road, and can reduce speed accordingly.

However, the only purpose of this reliance on the machine learning classification should be to improve the comfort of the drive (e.g., avoid abrupt braking). I believe we can reasonably expect that within reason, the self-driving car nevertheless maintains an absolute safety guarantee (i.e., it doesn't run into an obstacle). I say "within reason", because of course if a person jumps in front of a fast moving car, there is no way the car can react. I think it is highly unlikely that this is what happened in the accident -- pedestrians typically exercise reasonable precautions when causing the road.

[1] https://www.cs.cmu.edu/~zkolter/pubs/levinson-iv2011.pdf

> I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road.

Are you working on the next season of Black Mirror?

In all seriousness, my fear (and maybe not fear, maybe it's happy expectation in light of the nightmare scenarios) is that if a couple of the "weird and terrifying" accidents happen, the gov't would shut down self-driving car usage immediately.

>So we will have overall a much smaller number of deaths caused by self driving cars

Why? This is what the self-driving cars industry insists on, but has nowhere near been proven (only BS stats, under ideal conditions, no rain, no snow, selected roads, etc -- and those as reported by the companies itself).

I can very well imagine a greater than average human driving AI. But I can also imagine being able to write it anytime soon not being a law of nature.

It might take decades or centuries to get out of some local maxima.

General AI research had also promised the moon once again in the 60s and 70s, and it all died with little to show of in the 80s. It was always "a few years down the line".

I'm not so certain that we're gonna get this good car AI anytime soon.

Or take a slow approach like Google Waymo which was almost perfect so far on the roads. Uber is rushing it, and this could cost whole industry.

Seems like you're jumping to conclusions here. Let's wait to see what exactly happened. I highly doubt that any of these companies just use "straight" ML. For complex applications, there's generally a combination of rules-based algorithm and statistical ML based ones applied to solve problems. So to simply: highly suspect predictions aren't just blinded followed.

Totally agree on your premise that we can rationalize humans killing humans - but we cannot do so with machines killing humans.

If self-driving cars really are safer in the long-run for drivers and pedestrians - maybe what people need is a better grasp on probability and statistics? And self-driving car companies need to show and publicize the data that backs this claim up to win the trust of the population.

> These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention".

Tangent: is there a land vehicle designed for redundant control, the way planes are? I've always wondered how many accidents would have been prevented if there were classes of vehicles (e.g. large trucks) that required two drivers, where control could be transferred (either by push or pull) between the "pilot" and "copilot" of the vehicle. Like a driving-school car, but where both drivers are assumed equally fallible.

> This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one.

Do we even know yet what's happened?

It seems rather in bad taste to take someones death, not know the circumstances then wax lyrical about how it matches what you'd expect.

May be for now, autonomous driving should be limited to freeways and roads that dont have pedestrian crossings/pavements.

Another quirk of people, particularly acting via "People in Positions of Authority" is that they will need to do something to prevent next time.

Why did this happen? What steps have we make sure it will never happen again? These are both methods of analysing & fixing problems and methods of preserving a decision making authority. Sometimes this degrades into a cynical "something must be done" for the sake of doing, but... it's not all (or even mostly) cynical. It just feels wrong going forward without correction, and we won't tolerate this from our decision makers. EVen if we will, they will assume (out of habit) that we won't

We can't know how this happened. There is nothing to do. ..and.. this will happen again, but at a rate lower than human driver's more less opaque accidents.... I'm not sure how that works as an alternative finding out what went wrong and doing something.

Your comment is easily translated into "you knew there was a glitch in the software, but you let this happen anyway." Something will need to be done.

> So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails.

One of the big questions I have about autonomous driving is if it's really a better solution to the problems it's meant to solve than more public transportation.

Do you have any experience developing autonomous driving algorithms? Because you are making a lot of broad claims about their characteristics that only someone with a fairly deep level of expertise could speculate about.

Interesting insight. While self-driving cars should reduce the number of accidents, there are going to be some subset of people who are excellent drivers for which self-driving cars will increase their accident rates (for example, the kind of person who stays home when its icy, but decides that their new self-driving car can cope with the conditions).

Your comment reminded me of the One Pixel Attack [1] and my joke about wearing a giant yellow square costume...

[1] https://github.com/Hyperparticle/one-pixel-attack-keras

> Moreover, the human brain won't like processing these freak accidents.

I think this is really key. The ability to put the blame on something tangible, like the mistakes of another person, somehow allows for more closure than if it was a random technical failure.

Very well put.

It boggles my mind that a forum full of computer programmers can look at autonomous cars and think "this is a good idea".

They are either delusional and think their code is a gift to humanity or they haven't put much thought into it.

    I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road

Shouldn't be an issue with 3D cameras

To put this accident in perspective, Uber self-driving cars totaled about 2 to 3 millions miles, while the fatality rate on US roads is approximately 1.18 deaths per 100 millions miles [1].

[1] https://www.nhtsa.gov/press-releases/usdot-releases-2016-fat...

The appropriate comparison would be to ask how many pedestrians were struck by a car and killed.

Considering that humans would likely slow down if they see a pedestrian - even if one appeared suddenly - this is even more disconcerting.

This Rand study looks at impact of lives saved and recommends that "Results Suggest That More Lives Will Be Saved the Sooner HAVs Are Deployed". Any mishap, while most unfortunate and tragic for everyone concerned, should not result in kneejerk reactions!

https://www.rand.org/pubs/research_reports/RR2150.html

That’s not apples to apples. How many cars account for the nhtsa number and how many for Uber’s program?

Knowing that makes this much more infuriating.

Waymo has driven 2.7 billion miles in simulation [1]. Is that enough to give them some idea of how many deaths to expect?

[1] https://waymo.com/ontheroad/

that doesn't sound like a fair comparison. How many human interventions were there in the 2-3 million Uber miles?

The people leading the development should demonstrate that they can stop for pedestrians by personally jumping out in front of them on a closed test road. If they're not able to demonstrate this, they shouldn't be putting them on public roads.

Self driving cars are still subject to the laws of physics... unless you're going to dictate that self-driving cars never go above 15mph, I wouldn't advocate jumping in front of even a "perfect" self-driving car.

Braking distance (without including any decision time) for a 15mph car is 11 ft, for a 30mph is 45 ft. Self driving cars won't change these limits. (well, they may be a little better than humans at maximizing braking power through threshold braking on all 4 wheels, but it won't be dramatically different)

So even with perfect reaction times, it will still be possible for a self-driving car to hit a human who enters its path unexpectedly.

The ancient Romans would have the civil engineer stand under the bridge they'd just built while a Legion marched over it. That's why Roman structures are still around today!

"The people leading the development should demonstrate that they can stop for pedestrians by personally jumping out in front of them on a closed test road. If they're not able to demonstrate this, they shouldn't be putting them on public roads."

Although the actual logistics of your proposal might be challenging (child comments point out that some speeds/distances might be impossible to solve) your instinct is a correct one: the people designing and deploying these solutions need to have skin in the game.

I don't think truly autonomous cars are possible to deploy safely with our current level of technology but if I did ... I would want to see their family and their children driving in, and walking around, these cars before we consider wide adoption.

My opinion is that we will come to a point where self-driving cars are demonstrably, but only marginally, safer road users than humans.

From an ethical standpoint the interesting phase will only start then. It‘s one thing to bring a small fleet of high tech (e.g. having LIDAR) vehicles to the road. It‘s another to bring that technology to a saturated mass market which is primarily cost driven. Yes, I assume self-driving cars will eventually compete with human driven ones.

Will we, as a society, accept some increase in traffic fatalities in return for considerable savings that self-driving cars will bring?

Will you or me as an individual accept a slightly higher risk in exchange for tangible time savings?

This isn't a good argument because it implies if these AVs have successfully not killed their CEOs during a closed (i.e. controlled) test, that they are safe on public roads. But it seems like the majority of AV accidents so far involve unpredictable and uncontrolled conditions.

IOW, setting this up as some kind of quality standard gives unjustified cover ("Hey, our own CEO risked his life to prove the car was safe!") if AVs fail on the open road, because the requirements of open and closed tests are so different.

IIRC the people who programmed the first auto pilot for a major airliner were required to be on board the first test flight, so I have to think their testing methodology was pretty meticulous.

Reminds me of this: https://www.youtube.com/watch?v=Sr1ym96abkg

> The people leading the development of these horse-drawn carriages should demonstrate that they can stop for pedestrians by jumping in front of them on a closed test dirt-path. If they're not able to demonstrate this, they shouldn't be putting them on public carriageways

Sounds silly when compared against old tech.

Accidents happen, best we can do is try to prevent them.

This reminds me of a story I heard in college... an owner of a company that builds table saws demonstrated his saw's safety feature - a killswitch that pulls the blade down into the machine as soon as it detects flesh - by putting his hand on it while it was spinning.

very good point, I would not be surprised if Uber put those cars out with a barely working model to collect enough training data, the human operator to correct such errors was tired and didn't intervene. Some basic driving test for self driving cars or other mitigating factors need to be added IMMEDIATELY otherwise tons more will probably die or be injured. Train operators need to prove they are awake and with attention using some button presses, similar things need to be required for those research vehicle if you want to allow them at all.

human driven cars are currently, as we speak running people over on the streets and they have human drivers who don't particularly want to run over other humans. It was inevitable that this happened, and no matter how many people self driving cars run over will be worth it, since it will still be less than what cars are currently doing in terms of death toll.

There wouldn’t be many human drivers on the road if they had to pass this test as well.

Bad idea. Even Uber corporate VPs shouldn't be exposed to unnecessary injury risk.

https://www.youtube.com/watch?v=_47utWAoupo

goes to show, sdv's have to be perfect in the eyes of the public. you wouldn't seriously recommend adding a test like that to a regular driving licence.

also that testing does happen in the case of every av program i know of. closed obstacle courses with crap that pops out forcing the cars to react. look up gomentum station.

† i did it for you. http://gomentumstation.net/

This was a cyclist. So they should be on bikes.

Just like they take the repair mechanics on the first test flight after a major repair of a big airplane.

Said that few times: how does any AI recognize whether object in front of me is just a white plastic bag rolling in the street, or a baby that rolled out of crib. AI cannot know and will never know. And we cannot have cars drive smoth in traffic if every self driven car will stop before hitting an empty plastic bag.

Why? It’s not a problem if Uber kills pedestrians, even in situations where it’s completely avoidable. It’s only (legally) a problem if they’re violating the rules of the road while doing so.

This opens up an interesting question going forward. We can't rely on Uber themselves to analyse the telemetry data and come to a conclusion, they're biased. So really, we need self driving car companies to turn over accident telemetry data to the police. But the police are ill equipped to process that data.

We need law enforcement to be able to keep pace with advances in technology and let's face it, they're not going to have the money to employ data analysts with the required skills. Do we need a national body for this? Is there any hope of a Republican government spending the required money to do so? (no)

NTSB has already announced that it will investigate: https://techcrunch.com/2018/03/19/ubers-fatal-self-driving-c...

I guess we'll find out how competent they are at this, but I think they did a surprisingly good job with the previous Tesla investigation: https://dms.ntsb.gov/pubdms/search/hitlist.cfm?docketID=5998...

On the one hand I'd say give it to the NTSB, because historically they are really good at this sort of thing.

On the other hand, I'd wonder if increasing NTSB scope this much would drastically decrease the average quality of NTSB's work. Scaling ain't easy.

Aren't you basically just proposing that the NTSB analyze automobile telemetry, the same way they analyze aircraft telemetry data? Doesn't seem wildly outside the realm of possibility.

Law enforcement already conducts traffic accident investigations where the involved drivers are biased parties of inconsistent honesty and imperfect memory that can and do both accidentally and intentionally misrepresent the facts.

I don't think self-driving cars and their sensor data, even if they rely on the operator to explain what the car “remembers”, fundamentally shift the landscape.

That sounds pretty much like how things are done as it is.

For instance, take a look at:

https://www.ntsb.gov/investigations/AccidentReports/Pages/HW...

"The Tesla was equipped with multiple electronic systems capable of recording and transmitting vehicle performance data. NTSB investigators will continue to collect and analyze these data, and use it along with other information collected during the investigation in evaluating the crash events."

Don't we already have a huge infrastructure of police, the courts, and insurance companies in place to decide these very things everyday?

I mean, how is this different then all of the other accidents that occur every day? Yes a self-driving car is involved, but do people really think autonomous cars aren't going to be involved in fatal accidents?

Of course they are...but I've always thought that autonomous vehicles only have to be like 10% safer for them to make tons of sense to replace human drivers.

This sounds like something the NHTSA should do.

> We need law enforcement to be able to keep pace with advances in technology

Agree, Kumail Nanjiani (comedian) has a great rant on twitter about exactly this, ethical implications of tech-

> As a cast member on a show about tech, our job entails visiting tech companies/conferences etc. We meet ppl eager to show off new tech. Often we'll see tech that is scary. I don't mean weapons etc. I mean altering video, tech that violates privacy, stuff w obv ethical issues. And we'll bring up our concerns to them. We are realizing that ZERO consideration seems to be given to the ethical implications of tech. They don't even have a pat rehearsed answer. They are shocked at being asked. Which means nobody is asking those questions. "We're not making it for that reason but the way ppl choose to use it isn't our fault. Safeguard will develop." But tech is moving so fast. That there is no way humanity or laws can keep up. We don't even know how to deal with open death threats online. Only "Can we do this?" Never "should we do this? We've seen that same blasé attitude in how Twitter or Facebook deal w abuse/fake news. Tech has the capacity to destroy us. We see the negative effect of social media. & no ethical considerations are going into dev of tech.You can't put this stuff back in the box. Once it's out there, it's out there. And there are no guardians. It's terrifying. The end. https://twitter.com/kumailn/status/925828976882282496?lang=e...

It is scary, big tech orgs have no incentive or motivation to even consider ethical implications, whats worse is the American consumer has shown repeatedly that you're OK to do really shady stuff and as long as it means a lower price product/service for the consumer. We're in a kind of dark age of tech regulation and he's right it is terrifying.

Maybe something like the IIHS ( https://en.wikipedia.org/wiki/Insurance_Institute_for_Highwa... ), which is funded by insurance companies?

Though that would of course depend on how insurance even looks for SDCs. Maybe big companies like Uber will self-insure.

Perhaps a company like open ai or some other company setup by all the players involved could run the analysis in cases like this.

NTSB? https://www.ntsb.gov

State governments, especially those authorizing self driving cars, could authorize this to some extent.

How about we wait for the problem to present it and actually cause harm before we throw law enforcement, government regulation, a regulatory body, etc at it?

Important and and missing:

"[Uber] said it had suspended testing of its self-driving cars in Tempe, Pittsburgh, San Francisco and Toronto"[1]

1: https://www.nytimes.com/2018/03/19/technology/uber-driverles...

that's SOP for every av program. have incident? ground the fleet. doesn't matter why, or who's at fault.

Again... 'Uber has temporarily suspended the testing of its self-driving cars following the crash of one of them in Tempe, Ariz. The ride-hailing company had also been conducting experiments in Pittsburgh and San Francisco.'

I was resumed shortly after.

27 Mar 2017 (1 year ago)

https://spectrum.ieee.org/cars-that-think/transportation/sel...

FTA:

"The Uber vehicle was reportedly driving early Monday when a woman walking outside of the crosswalk was struck.

...

Tempe Police says the vehicle was in autonomous mode at the time of the crash and a vehicle operator was also behind the wheel."

That's a very bad look; the whole point of self-driving cars is that they can react to unexpected circumstances much more quickly than a human operator, such as when someone walks out into the road. Sounds like Uber's platform may not be up to that standard yet, which makes me wonder why they're on public roads.

On the other hand, it sounds like it happened very recently; I guess we'll have to wait and see what happened.

> That's a very bad look; the whole point of self-driving cars is that they can react to unexpected circumstances much more quickly than a human operator, such as when someone walks out into the road.

Some of these accidents are unpreventable by the (autonomous) driver. If a pedestrian suddenly rushes out into the street or a cyclist swerves into your path, the deciding factor is often simply the coefficient of friction between your tyres and the road.

The autonomous vehicle and the human attendant might have made a glaring error, or they might have done everything correctly and still failed to prevent a fatality. It's far too early to say. It's undoubtedly a dent to the public image of autonomous vehicles, but hopefully the car's telemetry data will reveal whether this was a case of error, negligence or unavoidable tragedy.

>> Sounds like Uber's platform may not be up to that standard yet, which makes me wonder why they're on public roads.

They're on public roads because they've decided that deploying their tech is more important than safety.

I don't see any other explanation. We know that it's pretty much impossible to prove the safety of autonomous vehicles by driving them, so Uber (and almost everyone else) have decided that, well, they don't care. They'll deploy them anyway.

How do we know that? The report by RAND corporation:

https://www.rand.org/pubs/research_reports/RR1478.html

  Key Findings

  * Autonomous vehicles would have to be driven hundreds 
  of millions of miles and sometimes hundreds of billions 
  of miles to demonstrate their reliability in terms of 
  fatalities and injuries.

  * Under even aggressive testing assumptions, existing 
  fleets would take tens and sometimes hundreds of years 
  to drive these miles — an impossible proposition if the 
  aim is to demonstrate their performance prior to 
  releasing them on the roads for consumer use.

  * Therefore, at least for fatalities and injuries, 
  test- driving alone cannot provide sufficient evidence 
  for demonstrating autonomous vehicle safety.

The main problem with self-driving cars is that they can't "read" humans' body language. A human driver can see pedestrians and cyclists (and other cars) and have a rough idea of what they're likely to do in the next few seconds, i.e. the pedestrian leaning out on the crosswalk curb is likely to step into the road soon. Even a reaction time of milliseconds can't make up for the (currently exclusive) human ability to read other humans and prepare accordingly.

Could a human have reacted fast enough to stop for someone jumping out in front of them? If the person jumped out so fast that nobody could have possibly reacted in time, then it's not a stain on the technology- even with an instantaneous application of brakes, a car still takes a while to come to a stop. If the human jumped out ten seconds earlier and was waving her hands for help, then it's an issue.

"the whole point of self-driving cars is that they can react to unexpected circumstances much more quickly than a human"

This statement isn't really true. Do you think Uber is investing in this because it makes their passengers safer? No, they are pretty much immune to any car crashes with a human driver, the risk and insurance hit is assumed by the driver. They are doing this to save money. They don't have to pay a driver for each trip. The safety aspect is just a perk, but Uber will be pushing this full force as long as the self-driving cars are good enough.

I'm more concerned that Uber will "lose" the video evidence showing what kind of situation it was, and we'll never be able to know if a human would have had ample time to react.

I have seen no evidence that any autonomous vehicle currently deployed can react faster than an alert and aware human. The commentariat tends to imagine that they can, and it's certainly plausible that they may eventually be. But I've never seen anyone official claim it, and the cars certainly don't drive as though they can quickly understand and respond to situations.

> That's a very bad look; the whole point of self-driving cars is that they can react to unexpected circumstances much more quickly than a human operator, such as when someone walks out into the road.

No, the whole point of self-driving vehicles is that firms operating them can pay the cheaper capital and maintenance costs of the self-driving system rather than the labor cost of a driver.

I imagine these vehicles (especially test vehicles) are also recording regular video and therefore getting a clear picture of what happened should be straight forward.

I agree completely. One thing that is important to me is that the whole self-driving field will learn from every mistake. In other words, every self-driving car should only make the same mistake once, whereas with humans, each human has to learn only from their own mistakes.

> Sounds like Uber's platform may not be up to that standard yet, which makes me wonder why they're on public roads.

Who exactly do you think is granting Google, Uber, etc. approval for trials like this on public roads? It's going to be some bureocrat with zero ability to gauge what sort of safety standard these companies' projects have reached.

There are no standards here... what were you expecting would happen?

Not bad even a self driving car can stop in time for deer jumping out from behind tree/bush cover.

People still need to look both ways when coming out from behind cars.

I want more details.

Here's a times article you reminded me of that mentions Saab engineers putting on crash helmets and running the 9-5 into a concrete moose.

http://www.nytimes.com/1998/04/05/automobiles/behind-the-whe...

> Sounds like Uber's platform may not be up to that standard yet, which makes me wonder why they're on public roads.

Uber cutting corners and playing fast and loose with legislation? Unheard of!

Here's hoping they get hit with a massive, massive wrongful death lawsuit.

I'd also add the goal of self-driving cars is to decrease costs for ride-sharing, home-delivery companies, etc, and also to decrease congestion via coordination amongst autonomous vehicles.

It's an interesting dynamic. We want this tech to be much better than us terrible humans before we deploy it and anything like us terrible humans is not acceptable.

The first martyr of the AI age, I suppose.

> That's a very bad look; the whole point of self-driving cars is that they can react to unexpected circumstances much more quickly than a human operator, such as someone walking out into the road. Sounds like Uber's platform may not be up to that standard yet, which makes me wonder why they're on public roads.

That's an invalid conclusion to draw from this accident.

There were 34,439 fatal motor vehicle crashes in the United States in 2016 in which 37,461 deaths occurred. This resulted in 11.6 deaths per 100,000 people and 1.16 deaths per 100 million miles traveled.[1] As far as I know, the instance this article is about is the first death in an autonomous vehicle accident, meaning that all the 2016 accidents were humans driving.

Why is it that you see one death from an autonomous car and conclude that autonomous cars aren't ready to be driving, but you see 37,461 deaths from human drivers and don't conclude that humans aren't ready to be driving?

I admit that there just aren't enough autonomous cars on the road to prove conclusively that autonomous cars are safer than human-operated cars at this point. But there's absolutely no statistical evidence I know of that indicates the opposite.

[1] http://www.iihs.org/iihs/topics/t/general-statistics/fatalit...

EDIT: Let's be clear here: I'm not saying autonomous cars are safer than human drivers. I'm saying that one can't simply look at one death caused by an autonomous car and conclude that autonomous cars are less safe.

They will have video for sure if since they are testing, then we will see. Your statement about reaction time is assuming so many things, may as well declare the AI guilty now right?

If you're familiar with that part of ASU campus you honestly could have seen this coming. There are a few different self driving cars in Tempe (Waymo, Uber, GM...) and Uber drives by far the most aggressive. They drive on a pedestrian heavy road and drive faster than most of traffic. They accelerate rapidly when a light changes and can brake hard. There are always tons of pedestrians in the area and it isn't uncommon to almost get run offer when even crossing during the day.

I wonder if it's because Uber algorithms are trained using Uber driver data (taxi drivers tend to be very aggressive drivers).

This matches my experience driving elsewhere. You need to adapt to the context, the car should not drive the same way everywhere.

Obviously more information is needed, but I thought the entire point of having a driver behind the wheel is to manually intervene to prevent this very situation?

I'm very curious to see how they'll investigate this and who will be determined to be at fault (person behind the wheel or Uber). It will likely set a precedent.

Human beings simply cannot switch between "not focussed" and "in charge of a car, taken out of autonomous mode, and actively avoiding collision" fast enough to avoid most accidents, unfortunately. Neither can humans maintain the focus required to be ready to do that when 99.9% of the time they're not required to do anything.

Semi-autonomous cars have drawbacks.

> I'm very curious to see how they'll investigate this and who will be determined to be at fault (person behind the wheel or Uber). It will likely set a precedent.

It's also plausible that the pedestrian was at fault. Under some circumstances it would be impossible for a human driver to avoid collision with a pedestrian, this may be no different.

The only way this is a reasonable failure of both the computer and the human driver is if they both physically had no time to react. Maybe that was the case, maybe it wasn't.

It could have been the fault of the pedestrian. I'm not saying it is, just saying there is a third possibility.

If humans can't prevent themselves from having accidents 100% of the time when they're in total control of the vehicle, why should we expect they could prevent an accident 100% of the time when they're in partial control of the vehicle?

" Obviously more information is needed, but I thought the entire point of having a driver behind the wheel is to manually intervene to prevent this very situation?"

I think it's time to point out the obvious, and require that autonomous cars apply the brakes first, and THEN require driver intervention.

And that they be a whole lot quicker to err on the side of braking.

Cameras getting fuzzy ?

Slow down.

Your ML algorithms are showing lower confidence measures for how they classify nearby objects and trajectories ?

Slow down.

nearby vehicles slowing down and you don't know why?

Slow down.

This is inexcusable.

Here we go. We'll now have the first traffic fatality trial where it's not drivers-trying-drivers but people-trying-a-megacorp.

(Disclaimer: I'm a bike advocate, so I may have a different perspective on some of this than most.)

Our car-based transportation system is far and away the most dangerous thing any of us accept doing on a daily basis. 40,000 die a year.

But when cases come to court, everyone on the jury has in the back of their mind "that could have been me if I lost concentration at the wrong moment, or made one bad judgement, etc etc."

So penalties are comparatively light for traffic fatalities. Big punishments are only meted out if the case is so egregious -- repeated drug use, flagrantly reckless behavior -- that the jury can be convinced that the driver is different from them.

In other words, drivers don't get punished for doing something dangerous, because everybody on the road is doing something dangerous. They get punished for doing something more dangerous than the norm.

In this case, there's no question that the "driver" is different than the jury -- it's a computer. Now the symmetry that made jurors compare themselves to the accused is broken.

The result, and what self-driving car advocates don't get, is that self-driving cars don't just have to be safer than human drivers to be free of liability, they need to be safe period. In a trial, they don't benefit from the default "could have been me" defense.

That's a HUGE requirement. In fact, it's probably impossible with our current road system. It won't just take better self-driving cars, but better roads and a major cultural change in our attitudes about driving.

As a bike advocate, I welcome this shift, but I also see how deluded many of the current self-driving projects are. Software moves fast, but asphalt and mentalities move slow. We're not years away from a self-driving transportation system, we're decades.

And this trial is just the beginning of that long story.

Most drivers avoid the most serious penalties, and more importantly most victims are denied restitution, not because of sympathetic juries but because of insolvency.

The chance someone will cause a grave accident and the chance someone is insolvent are not independent variables.

You get some real horror stories in law school about people trapped in burning vehicles with no remedy for the surviving family.

https://en.m.wikipedia.org/wiki/World-Wide_Volkswagen_Corp._...

One way you can solve for this is strict products liability. Normally we ask if there's a defect in design or negligence. That produces costly litigation where a lay panel reviews a bunch of technical engineering documents with no special training, with highly varying outcomes. Instead, just have the manufacturer pay some statutory compensation whenever their product causes harm. The social harm of the thing will be borne by its purchasers.

It's hard to do that with car accidents though, because driver error contributes so much to outcomes, it's often weird to punish the company.

But we could vastly simplify the auto insurance system and improve the safety of these vehicles by forcing them to just warranty against harm once drivers are out as a factor. (Insofar as a warranty is a promise not to do harm with a fixed penalty up front based on breach.)

It would be fitting if such a statutory regime were the earliest primary laws about AI, because that would officially make a requirement that ai does not harm humans into literally the first law of robotics.

>Here we go. We'll now have the first traffic fatality trial where it's not drivers-trying-drivers but people-trying-a-megacorp.

On even the slimest cause, plaintiff's council always tries include the manufacturer in the case. Anyone with deep pockets that they can pull in.

I agree. People wish not to avoid death, but to lessen their fear of death.

Arguments about how relatively safe self-driving cars will be are beside the point. A self-driving car is something that can kill you. The fact that death by self-driving vehicle is especially rare makes it especially scary.

A low accident rate is not a good thing, psychologically. Only when a risk is comfortably common can we comfortably begin to ignore it.

There's a current trend for people to anthropomorphise cars, with the backing of the industry (two eyes/lights and a mount/grill on the front of "him/her"). Won't the industry go into overdrive on the anthropomorphism front to try and preserve this judicial empathy?

I'd be interested in hearing more about why you are a bike advocate?

It's a sad reality, but regardless of how many times per day a pedestrian is hit by a human-driven car, such incidents involving self-driving cars will be headline news for years to come.

Having a hard time sympathizing here. It's one thing to fear air travel based on the very few but very publicized plane incidents (considering all the data we have on the safety of air travel). It's another thing to hold these self-driving car companies accountable, considering a) the lack of data and history of such programs, and b) their touted benefit as a safer alternative to human-driven cars.

Wow. This is dark but it feels real to me.

I had been assuming that self driving cars would be a 'reset' for our decision to value cars more than the countless lives they end. Self driving cars could take the irrationality and emotion out of driving. It would be America's "stop the murder of children" moment (The Netherlands rallying cry to reform auto traffic in 1970's).

But now I see that it can, and probably will go the other way. These giant companies could push to further restrict the rights of cyclists/pedestrians around roadways, vilify the 'stupid' people who made a reckless decision that resulted in an accident, and push our country even further into car culture.

As they should: humans fail independently, but autonomous cars can fail systemically, including hacking. Bicycles are known to be tricky for autonomous cars.

Even if its provable that self driving cars are 10x safer, every accident of this unfortunate kind will be used against autonomous cars. I don't see how to overcome this -- I don't expect reason will overcome the perceived danger.

You're goddamn blind. Yes, of course it matters more if I'm killed by an AI machine than it does if I'm killed by another human being. This is a new development, humans killing humans are not. What if it was autonomous warfare? It is surely coming about in a very near shipment, SOON. The world and history are not static, and techno-utopia is big fat lie, and the idea that things just get better as time advances frontwards is plain infantile. Reality is way more fucked up than you can imagine.

There may be no company I trust less to be open and honest about this than Uber. The arms race between companies will likely keep the most important lessons from this event proprietary. I'm very much not looking forward to hearing execs answer for this as I'm sure it will have the same amount of humanity as the rest of their comms.

This is my thought as well. Compared with Waymo, Cruise/GM, Tesla, Lyft, and the many other start-ups in this space, I think Uber is the least ethically scrupulous. I also doubt their self driving tech is as advanced as many of the other players, and I wonder if those other companies would have been able to handle this situation.

It's fascinating that so many brilliant people in this thread can talk so casually about any single loss of life for the tech built by the very same people in this thread. I would be devastated if I learned my code ended even one person's life.

You're in a bubble if you think a world hyper connected to social media that's in the same world scared that autonomy is about to kill millions of jobs in one jab and would NOT over-react to a single accident.

From the article it's impossible to know if the cyclist was at fault, so I won't jump to conclusions, but we can't, especially as the creators of the future of autonomous vehicles talk so dissociatively about human life. Either we go out into the public roads with very high certainty that OUR software and OUR hardware won't kill someone, or we wait till we get there.

Public opinion on human driving is doing just fine with human drivers right now, so we can and should take our time to get to the future we want or risk a backlash and a guilty conscious.

> Public opinion on human driving is doing just fine with human drivers right now

There were over 40,000 motor vehicle deaths in 2016 in the US [1] - more than a hundred daily. While loss of human life is always a tragedy, individual fatalities are absolutely worth it if we can begin to reduce that count sooner than by proceeding with excessive caution.

It would be morally unacceptable to delay development merely to avoid a guilty conscience.

[1] http://www.nsc.org/NewsDocuments/2017/Fatality-estimates-Jun...

Generally speaking, it is easier to pinpoint X thing led to Y tragedy than to quantify the tragedies that would have happened without X. It is incredibly hard to show what did not happen but "should" have. Even if you can get it straight in your own mind, good luck proving it to the world.

People are living longer and child mortality is down. We don't really celebrate that. We just complain about overpopulation and how we are destroying the environment.

When Iraq lit oil wells on fire on its way out of Kuwait, they were expected to burn for months and be a global environmental disaster. When crack teams from around the world converged and put them out far faster than expected, it was not celebrated with the same degree of fervor that it had originally been decried as a disaster.

Y2K was also predicted to be a global catastrophe. It was quietly prevented and gets remembered as "Those fools who ever thought this was a big deal!"

If self driving cars could reduce mortality compared to current rates and we were confident of that, waiting until it is perfected before releasing it means accepting ongoing deaths that don't have to happen. It's not uncommon for people to feel that new tech needs to be perfected while missing the fact that it may be a big improvement over the current status quo without being perfected.

I don't really know anything about self driving cars. I have mixed feelings about replying to your comment because I don't actually know how I feel about this particular issue. But the thought process you are putting forth is both common and not very pragmatic.

I guess a pithy rebuttal would quote some saying about the perfect being the enemy of the good. But I think that leaves out a lot of important points.

Thank you for adding this perspective. Honestly I am not sure how to rationalize the dueling thoughts in my head.

On one hand, the "just one isn't so bad" argument is rational. In terms of fatalities per mile, self driving cars already seem to be safer than humans as a class. Therefore it would be wrong of me to say that this tech does not belong on the road when I believe I should be allowed to drive.

On the other hand, you said it best. If I ever wrote code running on a machine that killed somebody I don't know how I'd continue in my career. Hell, I work at Google very far from Waymo but if one of our cars struck and killed someone I'd feel some personal responsibility.

So ... I guess it's complicated.

> I would be devastated if I learned my code ended even one person's life.

I agree, but there is already a lot of software in a car that can kill people in case of an error. In particular the airbag have multiple sensors and a complex heuristic to decide when the have to open, which ones, full or partial strength, ... https://en.wikipedia.org/wiki/Airbag#Operation An airbag that doesn't open can kill someone. An airbag that open unnecessary can kill someone. From Wikipedia:

> From 1990 to 2000, the United States National Highway Traffic Safety Administration identified 175 fatalities caused by air bags. Most of these (104) have been children, while the rest were adults. About 3.3 million air bag deployments have occurred during that interval, and the agency estimates more than 6,377 lives saved and countless injuries prevented.

So, there is already a lot of programmers that have to be careful because an error can kill someone in the car (or the plane, or a medical machine, ...) I guess it's more easy to ignore the people killed by airbags because we are use to them and we imagine that they have a simple sensor (something like a light switch. So in case of a failure nobody is guilty.

With a self driving car, the first problem is that it is new and scary, and the second problem is that in an error that creates an accident we are use to blame the driver. But now we don't have anyone to blame (perhaps the programmer).

My career is in software, but when I peer out of my cave at those in more traditional engineering backgrounds, I realize they have so many more checkpoints and structures for guaranteeing performance / operability under load. For 99% of us in software, if our software fails, nothing really happens of consequence other than downtime - no one dies. In this case, the software written can have mortal consequences. Maybe there's a need for increased scrutiny on software development - especially as IoT gains more popularity and people are relying more and more on the software that supports or replaces human involvement in critical and/or potentially high risk activities.

Raises the question though: will stakeholders be willing to pay the upfront financial costs to obtain the level of quality and assurance that other engineering disciplines provide? We'll see.

What we can't do as the creators of the future of autonomous vehicles is give in for emotional appeals when the relevant thing is statistics about car accidents. If the media does it, bash the media (and hope for decent publications to support the appeal of relevant people in the field). If politicians do it, bash politicians, though nothing really helps when it comes to politics. But don't be nice to bureaucrats and their oblivious supporters who are actually preventing thousands of lives being saved every year by slowing down development and deployment.

>Either we go out into the public roads with very high certainty that OUR software and OUR hardware won't kill someone, or we wait till we get there.

As long as it kills fewer people than would otherwise have died due to negligent human drivers I'd say it's already there.

Crossing a 4 lane road (where speed limit is at least 40mph) outside crosswalk? It's clear that it is cyclists fault. The only thing that is questionable - would a human driver have avoided this or at least hit the cyclist with lesser speed.

As someone said on Twitter, among software developers, "ethics" is seen as a hobby at best.

>The self-driving Volvo SUV was outfitted with at least two video cameras, one facing forward toward the street, the other focused inside the car on the driver, Moir said in an interview.

>From viewing the videos, “it’s very clear it would have been difficult to avoid this collision in any kind of mode (autonomous or human-driven) based on how she came from the shadows right into the roadway,” Moir said. ( https://www.sfchronicle.com/business/article/Exclusive-Tempe... )

Not saying the software wasn't partly at fault but it doesn't look that clear cut. It sounds like better sensors could have helped.

Shouldn't '[Coming] from the shadows' be irrelevant when the sensors are LIDAR?

Is it normal for police to give these kind of statements right away? I thought they need to present evidence to the court of law instead of giving verdict themselves.

> Elaine Herzberg, 49, was walking her bicycle outside the crosswalk on a four-lane road in the Phoenix suburb of Tempe about 10 p.m.

> Tempe Police Chief Sylvia Moir said that from viewing videos taken from the vehicle “it’s very clear it would have been difficult to avoid this collision in any kind of mode (autonomous or human-driven) based on how she came from the shadows right into the roadway."

This seems like an unfortunate accident, but it's not at all clear that the car was at fault or that it could have done anything to prevent it. There was even a human behind the wheel and they didn't react in time either.

I can't describe how shocking it is watching many HN commented blame a person killed by a machine and defend the thing instead. There are even those who are saying that such deaths are necessary and useful for the advancement of technology.

I see the Waymo cars while I walk to lunch (AZ) quite often. One time while walking through the crosswalk a Waymo turning right cut me off as I started into the crosswalk despite my having the walk indicator. This seems to be a very difficult scenario for a self-driving car as there are often people at the corner that are either crossing the other direction or just waiting on the corner. I sent them a note via their feedback link, but never heard back.

This is a non trivial situation for human drivers, too, when I am on a bicycle or driving a car I have to wait and look at the pedestrian until I am sure of their intention. That seems like a very difficult problem in that a lot of times people are on their phone or watching a child or dog, I often have to slow to a standstill and speak to people to ask them what they are planning on doing.

The last time I visited Phoenix, I recall being shocked at far apart traffic lights were in the city. Between the Wal-marts and the giant four-to-six bedroom homes, it might be normal to see a major street have a mile between traffic lights (and presumably, crosswalks). It was very different from, say, driving in New York, San Francisco or Seattle.

I'd be curious whether "walking outside of the crosswalk" turns out to be material in the analysis by NHTSA/NTSB. Were the crosswalks big enough and close enough that a pedestrian would have normally walked inside one? Was the interchange a regular 90-degree crossing of two roads, or was it something more irregularly shaped? What is the annual rate at which pedestrians get hit at this intersection by human drivers? Do self-driving cars need more examples of "pedestrians walking outside the crosswalk" in their training sets?

I hope they aren't expecting people to only appear at intersections:

"The California Vehicle Code says you can actually cross any street as long as you aren't a hazard to vehicles. It is also legal to cross mid-block when you're not between two intersections with signals." [1]

The article points out that LA has a different rule. Uber's going to need to geofence locations and use different rulesets for different cities.

[1] https://www.scpr.org/news/2015/04/14/50992/how-do-you-cross-...

Former Lyft eng here. From my vantage point, as an industry, we're nowhere near where we should be on the safety side. The tech companies are developing driving tech privately instead of openly. Why can a private for profit company "test" their systems on the public roads? The public is at serious risk of getting run over by hacked and buggy guidance / decision system. Even when a human operator has his hands hovering 1 inch off the steering wheel and his foot on the brake, if the car decides to gas it and swerve into a person, it is probably too late for the human crash test driver to overtake. This is going to keep happening. The counterargument FOR this is that it is overall a good idea for the transportation system if the number of crashes & deaths is statistically less than human operated cars. I see this as the collision of what's possible with what's feasible and that we are years away from any of this being close to a good idea. :( Very sad for the family and friends.

Perhaps companies need to test their safety devices first. I.e., first prove that their LiDAR correctly identifies pedestrians, cyclists, etc. From there, build test-vehicles with redundancy, e.g. with multiple LiDAR devices. Then prove that the vehicles actually stop in case of emergency. And only then actually hit the road.

Of course, the US department of transportation should have set up proper certification for all of this. They could have easily done so because they can arbitrarily choose the certification costs.

Even worse: There are people who test their DIY self-driving hacks on public roads:

https://youtu.be/GzrHNI6eCHo?t=100

Using https://github.com/commaai/openpilot , which is cool but not on public roads.

> Why can a private for profit company "test" their systems on the public roads?

I mean, this is the way cars have always operated. It is somehow any different to let them "test" there brake system, there acceleration systems? Car companies have always been able to do whatever they want, who cares about the deaths.

For some reason, when it comes to roads, we all just accept a certain amount of deaths each year. Over a million people are killed every year around the world, up to 40,000 in the US alone. That is acceptable according to every car driver.

The FLOSS side has argued for openness in critical systems for ages. Medical devices such as pacemakers, air plane systems, automatic breaks on cars, voting machines, and the list just goes on. If the device get hacked or has any bugs then people will likely die (except for the voting machine), but the code is kept private even for those running in devices which literally gets operated into your chest and which a person has to trust with their own life. Governments all over the world seems to be very much against the idea to require that such critical code is made openly for the sake of safety, as that would leak the oh so important trade secrets.

The Washington Post has an updated story with the victim's name:

https://www.washingtonpost.com/news/dr-gridlock/wp/2018/03/1...

> Police said the vehicle was northbound on Curry Road when a woman, identified as 49-year-old Elaine Herzberg, crossing from the west side of street, was struck. She died at a hospital, the department said.

No mention of her being a bicyclist. My gut instinct is to accept that the victim was indeed a pedestrian -- with the assumption that they'd have that detail cleared up by the time they have her identity ready to release to the public.

FTA "Elaine Herzberg, 49, was walking her bicycle outside the crosswalk".

The article says "as soon as she walked into the lane of traffic she was struck".

The police sergeant also "said he believed Herzberg may have been homeless".

These three things may together indicate that even a human driver might have struck and killed the lady if she entered into the roadway unsafely. I'm not sure how everyone (especially on Twitter) can be jumping all over Uber yet until the investigation is complete.

Sad of course, either way, but not yet enough evidence to assume that the algorithm+driver combo was any less safe than a human driver alone.

The second ever [1] fatal car crash involving a self-driving car. This is the first one where a third party is killed.

[1] https://www.theguardian.com/technology/2016/jul/01/tesla-dri...

It's terrible when anyone is killed in an automobile accident. The irony is in the promise of autonomous vehicles to prevent this very thing.

I've heard the general report, and I find it interesting that a supervisor was at the wheel, and they, as well as the car, failed to prevent the pedestrian from getting hit. I read they were not in a crosswalk and I'm sure the specific details of the accident matter greatly. It makes me wonder if a human driver with no autonomous system would have faired any better since someone paid to be alert and a car designed to be alert both failed to prevent the accident.

I'll reserve final judgment until more details are out.

In the early 20th century the automobile industry mobilized to downplay the deadly threat automobiles posed to pedestrians, and lobbied to create new laws to frame pedestrians as being at fault instead of car drivers (eg. jaywalking).

I expect we'll soon see tech companies do the same in order to favour their autonomous automobile businesses at the expense of pedestrians and cyclists.

Can we please not overreact and send the whole self-driving research down the drain?

What does the statistics say? How many miles have the self-driving cars driven, and how many deaths were they responsible for? How does it compare to a human driver?

When a million humans drive a car for a mile, and 1 of them results in a death, it's easy to pinpoint the blame on a "random drunk/distracted driver". How'bout start thinking of the self-driving software as a combination of all human drivers, with just a much smaller odds of having a drunk/sun-blinded/distracted driver than an average human driver.

According to the latest info, it doesn't look like the system was at fault at all. Seem like the likely homeless woman abruptly stepped into traffic. Every accident still of-course is terrible, but some maybe are unavoidable. https://www.sfchronicle.com/business/article/Exclusive-Tempe...

I know this is a slightly insensitive question to ask right now, but assuming that what happened was that the car tried to turn right and the cyclist came from behind and crossed into its path[0]: Who would actually have been in the wrong?

I cycle along paths like this from time to time myself, and I always assume that I should let a turning car pass in front of me. As soon as I see the turn signal I will either fall back or even pass into the car lane to overtake on the other side.

Of course that's in part because I know I would lose the fight anyway, but also because I think I am not actually supposed to be "overtaking" them on the right side, so they have the right of way before me.

What do the rules say in this case?

[0] This is based on this link posted elsewhere: https://www.google.com/maps/@33.4370667,-111.9430321,3a,75y,...

Not to dismiss the tragedy of this incident, but it should be expected that self-driving cars kill pedestrians -- just at a rate lower than what's expected from human drivers.

Perhaps there's a better metric to look at, but I'd like to see number of deaths caused per miles driven.

If Uber's self-driving cars are killing more pedestrians than a human driver would, we have a huge problem, but I'd be willing to bet they're at least an order of magnitude safer in this respect.

Self driving cars are one of the things where we absolutely cannot afford to do "move fast and break things".

If there was any whiff of shortcutting, poeple need to go to jail and Uber should be bankrupted by penalties and lawsuits.

My understanding, from talking to someone who works for the NTSB, is that the software controlling autonomous vehicles is currently completely unregulated, and that companies like Uber, Tesla, etc. have refused to provide access to their software for external review.

Recently I've seen a lot of comments defending autonomous vehicles from a statistical standpoint, but even if these cars have the potential to be safer on average, it scares me to think that the software driving a car is less regulated than its airbags or seatbelts. Especially considering the auto-industry's tenancy to ignore safety issues without external pressure [0].

Is my information outdated? Have there been efforts to review or regulate autonomous driving software in the last year?

[0] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed

Unless this was a situation that was impossible to anticipate and avoid, this looks bad. It would mean that both, the autonomous driving system and the safety driver failed.

Not hitting pedestrians should be close to the top in priority, so while failures of the software are certainly expected during this kind of test, this is one of the worst kinds of failures that should have received a lot of attention before ever putting the vehicles on a public road.

The safety driver failing makes me wonder how well qualified they are, or if there is an issue with staying alert for long times without actually driving the car.

The video says bicyclist and shows a bent bicycle. So pedestrian or bicyclist?

What happens when a self driving car in the future from a service is driving empty back to some parking? And hits someone? Does it call ambulance or police?

While incredibly sad, this incident is not terribly surprising because the technology is nowhere near safe enough.

Unfortunately, this will inevitably set the field back by at least 10 years with stricter regulations. We've seen the same with pharma and aviation, where innovation was slowed drastically and is only realistic for big companies. This is especially true here because if self-driving vehicles get banned from public roads, companies like Waymo will be forced to develop bigger private "test tracks" that approximate cities to have any hope of building deployable technology, and they will face the same fundamental limitation – the data will come from an approximation/model and not the actual location with real-life scenarios.

Although there's no doubt that regulations are crucial to prevent such mishaps, I do hope lawmakers can find the right equilibrium on the tradeoff between safety and innovation.

Statistically speaking, I am curious to know the data on these areas: - number of miles driven per fatal accident on self-driving - number of miles driven per fatal accident on a human driver.

It is important to know whether our machine is now doing worse/better than our counter-part and this means a lot for our self-driving initiatives.

This was inevitable. Some of the developers for self-driving vehicles are immensely careful, some are not. And the regulatory oversight is currently not sufficient to allow the one to operate but not the other.

The reason self-driving car companies have flocked to Arizona is that Arizona chose to pretty much completely deregulate them: They don't even have to report statistics to the state like they do in California.

It's unfortunate, but unsurprising, the first pedestrian killed by an experimental self-driving car allowed on public roads was in Arizona.

1500 comments before knowing the real reasons. It could be suicide. The dead woman has history of substance abuse and arrested 6 times. http://www.dailymail.co.uk/news/article-5519433/Self-driving... http://fortune.com/2018/03/19/uber-self-driving-car-crash/

It seems like a bunch of what-ifs that normally come up with self-driving cars are about to get answered and precedents are about to be set.

I assume this case will also be one of the most well-recorded cases of a fatal car accident in history as well, given the amount of sensors and equiment on-board a self-driving car, along with eye witness testimony from the operator on board.

Can't tell if Uber has just been incredibly unlucky as of late or if just enough of their employee-base is incompetent as to prevent them from just having a quiet year with no large failures.

Hype cycles are beginning to cause loss of life. Self driving cars are being allowed to operate at a far faster rate than other transportation innovations would have. This is because of the enthusiasm around technology. We need to realize the difference between social media apps and steel.

Had the car passed the driving test?

There should probably be a specific, complex and comprehensive test for autonomous vehicles. Also I'd want to see shared liability, the company are driving it (by proxy through their software) so they should be liable to some extent.

My personal hunch is that fully autonomous self-driving tech is not theoretically possible under currently known computational models, because it implies many well-known NP-Hard problems. Self-driving companies are betting on the ability to find a heuristic/approximation that works "sufficiently well". But I strongly feel that the chasm that needs to be crossed to be "sufficiently good" is not one of magnitude (i.e. we just need more testing!), but of theoretical boundaries, due to the existence of at-least two sub-problems which are not computationally solvable: 1. prediction of what pedestrians/cyclists will do next, and 2. accounting for sensor input distortion under bad weather conditions.

Humans can solve these problems due to life experience, not just driving experience. In other words, I think we're gonna need fully-conscious AI to solve self-driving.

The only way self-driving tech will reach production is if the input space is restricted, which is a significant-but-not-groundbreaking iteration on what we've been doing for decades with airplane autopilots and self-driving monorails. Sure, we can have self-driving cars on specifically designed freeways, but nothing more.

Sorry, but what's the point of this?

> Elcock said he believed Herzberg may have been homeless.

I do not think that Herzberg possibly being homeless adds any meaningful information. On the contrary, I feel this may subtly support unconscious prejudices.

Curious how the numbers work out of number of miles [1] to number of deaths of autonomous vs standard vehicles.

I know the scale is different but since this is the first death I’m curious if the percents fall in line.

[1] is distance the right metric?

Why is Uber allowed to test its software for free on public roads ? Why doesn't it use a purpose-built test-track of its own, where it must prove that its software won't kill people (using robotic crash-test dummies to avoid) before it is allowed to use public roads ?

I just don't trust Uber and its engineers enough to trust them as a self-driving car company. Even before this incident.

Their organisation has been shown to be deceitful, their CEO to be devious. I can completely believe they would cut corners or claim safety where they know there is risk. To get to market ahead of the competition knowing that the fate of road users lay in a dice roll.

Technically they had to resort to stealing Waymo's IP to get ahead, and their self-driving cars have previously been shown to behave recklessly.

I do, tentatively, trust Waymo. I've seen enough about the resources they've put into this, the extensive testing they've done, and their safety record so far to at least give them a shot.

Why am I not surprised that this happend with an Uber self driving car?

In fact, their culture of ignoring rules and common sense might be okay for business development, but with respect to human safety it is just irresponsible and inhuman when it comes to self driving cars.

It's not hard to believe that self-driving autonomous cars will make the roads safer overall.

The issue is that while fewer lives will be lost, it will be a different set of lives than would have been lost without self-driving technology.

Those lives will have their day in court.

> Car was autonomous with driver behind wheel

Let me make a prediction. Uber will claim that the driver was in control of the vehicle at the moment of the accident. And the "driver" will receive a bonus payment in the coming months.

This paradigm-shift is a chance to address one of the leading causes of preventable death in the world.

It doesn't surprise me for a moment that this was Uber (though it might have been a Tesla). From all I've seen and read, these companies are racing unscrupulously, and some have inferior technology compared with others.

Waymo seem to be approaching the problem from a safety direction rather than a pure race for profit.

We need to hold SDs to a higher standard than we hold human drivers. Not 1.5x the standard; 1000. 10,000. 100,000.

And every death must be treated as a serious failure of engineering. These are preventable deaths.

I know quite a people working on this at Uber ATG. They are in panic mode.

Really? This is a completely new tech, and process. How is any reaction "a standard move?" I guess it could be standard for any new tech to result in a death from autonomous machinery. In any case, things like this are why I don't think fully autonomous is nearly ready in city environments. I'm not saying it isn't close, and shouldn't be worked on. I just don't think the intelligence systems are near as worked out as they should be. Not to mention the level of sensors needed.

Given how incompetent Uber is at regulation of anything else in their company, is this really a surprise?

A while back I was riding my bicycle home from work and saw one of the Uber robocars coming the other direction while I was making a left turn so decided to test their reaction algorithm by starting my turn while they were still approaching -- car didn't even flinch while a human driver would've at the very least honked at me.

As an aside, I've also been hit in a bike lane in Tempe by a human driver unfortunately, if it were a robocar I'd be living on the beach somewhere off the settlement check.

I would hate to have been the (backup driver) in this sad situation. Regardless, the regulations couldn't come soon enough.

From SF Chronicle [1]:

<QUOTE> The self-driving Volvo SUV was outfitted with at least two video cameras, one facing forward toward the street, the other focused inside the car on the driver, Moir said in an interview.

From viewing the videos, “it’s very clear it would have been difficult to avoid this collision in any kind of mode (autonomous or human-driven) based on how she came from the shadows right into the roadway,” Moir said. The police have not released the videos.

The incident happened within perhaps 100 yards of a crosswalk, Moir said. “It is dangerous to cross roadways in the evening hour when well-illuminated, managed crosswalks are available,” she said.

...

“I suspect preliminarily it appears that the Uber would likely not be at fault in this accident, either,” Moir said.

However, if Uber is found responsible, that could open a legal quagmire.

“I won’t rule out the potential to file charges against the (backup driver) in the Uber vehicle,” Moir said.

But if the robot car itself were found at fault? “This is really new ground we’re venturing into,” she said. </QUOTE>

[1] https://www.sfchronicle.com/business/article/Exclusive-Tempe...

Forgive my ignorance, but isn't this not supposed to happen with LIDAR equipped cars?

My understanding is that a LIDAR provides a completely accurate 3D map of the surrounding area around a self driving car. This is in contrast to Tesla's image recognition approach which makes a 3D map from 2D images. So is seems like a pretty giant bug if a self driving car ever crashes into another object with LIDAR, since it should always know the location of external objects.

Please correct me if I am wrong!

Something of a self-driver car Rorschach test. On the one hand there is the tragedy (car hits person, person dies), on the other there is the technology (computer doing the driving), and then there is the fog of reporting where actual data is hard to come by because people are people and report on the things they saw/heard that were important to them.

Last night there was a story on CBS about the first 'self driving bus' going live in San Ramon California. [1] Where the reporter steps out in front of it to see if it will stop (spoiler alert, it does).

And while it was a tragedy, its unfortunate that because it was a 'self driving' car this fatality gets more coverage than the thousands who are killed by 'human driven' vehicles. Bicycle advocacy groups have been arguing for years that better, separated, bike lanes would save lives. Perhaps the companies behind self driving can get behind that effort to protect bike riders from humans and make it easier on their cars.

[1] https://www.cbsnews.com/live/video/20180319105443-california...

So this is it then? First fatal accident involving a fully self-driving car which might have actually been the car's fault?

Well, it was obviously going to happen sooner or later. It'll be interesting to see what the fallout is. Up until now regulators have been, surprisingly, taking a pretty relaxed approach to regulation of self-driving cars. Hopefully this one accident doesn't impede development on other autonomous car programs too much.

Very Relevant video on what the discussions about 'the ethics of self driving cars' miss: https://www.youtube.com/watch?v=ozcaLnTuidU

Summary: We spend too much time talking about what decision the car should make in these situations, not enough on how allowing a corporation to make those life and death decisions changes our society.

Situations no self-driving car can avoid:

  - philosophical dilemma
  - physics-constrained reaction times
  - actions that violate the rules of the system

Solutions:

  - protected car lanes
  - protected bike lanes
  - protected pedestrian lanes

Reasons why these solutions are not put in place:

  - cost

Until humans determine that the cost of human life is higher than the cost of upgrading infrastructure, we should accept human death as a regular part of autonomous driving, just the same as we do for non-autonomous driving. 37k dead people every year in the US due to human drivers.

Top reasons for auto accidents today include inclement weather, reckless driving, speeding, driving under the influence, and distracted driving. In theory, most of those could be solved by autonomous driving. But then the list of reasons for accidents would change to whatever new reasons cause autonomous car accidents, such as damaged sensors, programming errors, equipment failure, road hazards, etc.

Even with autonomous cars, we will still need protected lanes, and we will still never implement them, because we don't really care when people we don't know die.

I know when Uber was stopped by CA DMV to operate its self driving cars without a permit, the Arizona governor went all out to promote the state as a beacon of business friendliness and deregulation etc. etc. Perhaps, when something is as critical as self-driving cars going around along with pedestrians and bikes and other vehicles, a more cautious and deliberate approach is warranted ?

I'm saddened by this incident but have thought a lot about this eventuality. there are a few levels of societal acceptance of self driving car death outcomes i can think of:

A - Human equivalent: self driving car obeys reasonable rules that a human also would eg minimum speed on a highway and kills someone

B - Trolley problem: (very artificial scenario) self driving car has to choose between killing N or M people where N > M and wrongly (in hindsight) chooses N

C - Car fault: self driving car kills someone in a situation where no human would have

As a society we would probably accept A easily but B starts to get shaky. C currently looks completely unacceptable, BUT I would argue that society has to get to a point where even C is ok conditional on the probable result that overall car deaths decline dramatically.

In other words we will have to get to a point where individual deaths are extremely regrettable but the overall death reduction of adopting self driving cars are so undeniable that the individual deaths can be discussed without also talking about banning SDCs in the same breath.

The self driving car craze baffles me. I would only trust it if it were on a road made for it and the cars communicated to one another.

There are only two paths from here - an escalation of the scramble to real world driving hidden under self-serving rationalisations, ("the bigger picture") or a step back to some industry self-regulation, where self-driving just does not happen till the sensors, the maps and the algorithms are much much much improved.

We know very little about the circumstances here, apart from the obvious tragedy, but given that we learnt from Tesla that you cannot rely on cameras alone, whatever is learnt from this must be shared amoung all the industry and serve as the baseline for future.

I would prefer an outright ban on driverless cars in real world settings (until perhaps each competing car can safely navigate the worst sensor data other companies can offer) but as a minimum we need a neutral clearing house similar to the airline industry.

There are on this forum enough industry insiders, plus a sprinkling of regulators that a global self-regulated path could easily be forged.

Given I've seen (just last year) an Uber self driving car run a red light in SF, nealy hitting several people - this is not at all supprising.

I do think Uber should be charged if the car is found at fault. Not a civil case, criminal. We know machine learning algorithms fail all the time (like this). It shouldn't be alright to just ignore this.

Of course it was Uber. I'm sure they don't even give a fuck any farther than the cost to their brand either.

Wow, this is sad.

I always imagine I would feel uncomfortable when developing software in certain areas such as this, or e.g. for space rockets containing humans. Errors have a much higher impact than when only developing web applications where the worst case scenario is a tempoarily unreachable website, or maybe some data loss.

I wonder how this will be treated legally.

Police can't just write "killed by python script" as a cause of death, that would be crazy, and there is no legal framework for cases like that. My guess is that A.I. could be equated to cruise-control, meaning person behind the wheel is responsible.

Maybe someone knows better?

>>The pedestrian was outside of the crosswalk. As soon as she walked into the lane of traffic she was struck.

>>Tempe Police Chief Sylvia Moir said that from viewing videos taken from the vehicle “it’s very clear it would have been difficult to avoid this collision in any kind of mode (autonomous or human-driven) based on how she came from the shadows right into the roadway.

A lot of comments on here are faulting machine learning for this or algorithmic flaws, but unless we see the video it is really hard to blame the algorithms. Based on the quote from the story, it sounds like someone stepped out into the street, not in the crosswalk, at night, directly in front of a car going 40 miles per hour. How could a human or any algorithm cope with that scenario?

My non-expert opinion is that we will not see self driving cars become reality in our lifetime. I don't understand why anyone believes training on some videos/pictures to calculate some probabilities would ever be enough to handle all the complexities of the real world.

I think it would be insane to expect 100% safety from any technology, let alone new technology. People do crazy things on the road, I am witnessing it almost daily. And if you look up dashcam videos on the internet, you'll be scared to ever drive again. Expecting that some magic technology can deal with all this craziness with 100% safety is impossible. And then we'll add bugs and malfunctions, which absolutely every technology has, especially - new one.

So, brace for the incoming wave of similar news. These things will happen, and these things will be heavily publicized. Hopefully it won't cause stupid overreaction to the tune of banning the technology altogether or surrounding it with a bubble of regulations so think as to make in infeasible.

That car has to have a dashcam -- what's the current caselaw with regards to the 5th amendment?

I love how everyone is throwing fuel on the fire here with no actual details as to what happened in the accident. It could simply be she ran out into oncoming traffic and got hit. Hard to say a human driver could have performed any better. The devil is in the details.

Why is Uber allowed on public roads?

I really don't understand the need to put a self driving car around pedestrians or other human drivers. As with any machine learning/control system, wild instabilities can and will crop up. Why don't we simplify the problem domain and keep well suited, designated infrastructure just for self driving vehicles, and solve the last mile problem with a slow 15mph creep to the ultimate destination. I mean, it's impressive that cars can drive themselves through a busy intersection alongside other human drivers, but it is by no means necessary, especially when adding humans into the mix adds a degree of uncertainty that is very difficult to account for...

> “The pedestrian was outside of the crosswalk. As soon as she walked into the lane of traffic she was struck”

Uber has some fucking shitty programmers guaranteed. They murdered this lady. Think about it:

Smart cars basically have 360 degrees of sensors. There’s no way it didn’t detect that lady coming up on the car and the fact that it kills her says to me there was no counter measure. It probably didn’t even slow down until the moment of impact. That's some fucking elementary school shit.

> Chance of Collision? 100%. Maintain Speed

In structural engineering if you designed a bridge poorly and it collapsed killing people you'd lose your ability to practice engineering. The same should be applied.

I wonder how companies handle these situations? Most places have clear requirements eg stop, check if anyone is injured, contact police, remove debris from road if safe to do so. These things aren’t so easy without a driver.

Based on landmarks from the photo from the news article, and the fact that the accident supposedly happened near Mill and Curry, I'm assuming that this was the location of the accident:

https://www.google.com/maps/place/N+Mill+Ave+%26+E+Curry+Rd,...

Uber must have a full video of this - just in case. I can hardly think they did not have a full record of this, as well as all miles traveled? It may be in these 1366+ comments, but I have not read them all to see.

I never understood the recent push towards self-driving cars, as it already seems perverse that the entire surface pavement has been dedicated primarily to motor vehicles and only secondarily to our actual selves, as pedestrians.

Putting further pressure on what should really be 2 discrete planes of activity (perhaps one in tunnels below ground or in the air, and I don't mean the pedestrians!)

Perhaps now the arrogance of such bad ideas becomes a little bit more clear to the tech set of folks, that in my view were excessively pushing such strange goals upon a public that wasn't clamoring for it.

To repeat my comment from a previous discussion which brought a lot of downvotes: what happens when (not if) a self-driving car runs-over and kills someone (e.g. because of a software bug)? Do such cases cause criminal penalties? Who is penalized? Or will all cases of autonomous car accidents with deaths become civil cases? If so - do human drivers get the same new rules or if they kill someone by accident (because they got distracted) they still go to jail? Is that fair?

In this particular case I assume the operator will be thrown under the bus, which is also unfair.

Why am I not surprised that, among all the teams competing in this space, many of which have clocked in far more miles than them, Uber is the first one to kill someone?

This may well just be bad luck. But I cannot shake the feeling that if Uber started an ice cream venture, they would store their molasses on a hill in a Boston. The only way to get “humanity” associated with Uber would involve an Uber zeppelin.

Or it’s a conspiracy, because nothing is as threatening to Uber than autonomous cars. This is certain to invite more regulatory scrutiny. Just kidding... I think..

Why is everyone concentrating on the software? There was a human in charge and the car was exceeding the speed limit. That sounds like either driving without due care and attention or reckless driving. After all the driver's reaction time is almost certain to be much longer when supervising an autonomous car than when driving directly making it much more important to adhere to the rules.

And if it can be shown that it was company policy to allow the cars to exceed the speed limit then heads should roll all the way to the top of the company.

Why not test these things on a private track? You have a multi-billion dollar company attacking this problem and it feels like all they tested before they hit the public roads was lefty and righty.

Looking further then the devastating tragedy at hand, it might be necessary to get the question of responsibility publicly answered. I dont think I am the only one here who has deliberately chosen not to work in fields where human lifes are at risk.

Despite the likely outcome, that the engineers behind this took every safety percussions, I think its extremely important, to get the message across, that people with an CS background with such use cases are going to be held to the same standard as for example mechanical and electrical engineers.

And yet (disclaimer - reflects only my personal view, might not correspond to the current state of affairs at Uber, it is possible that Uber cars are still safer than human drivers, but still - one who forgets the history tend to repeat it):

https://www.forbes.com/sites/samabuelsamid/2016/12/15/the-tr...

Let’s take the extreme example where a bug in a self driving semi truck leads to it mowing down 50 people on a busy sidewalk. How are the victims to get justice? In almost all incidents besides a human truck driver having a major health incident, that driver would go to jail but it seems that there will be little repercussions beyond fines for autonomous driving companies. I think once a mass fatality occurs people will scream for autonomous driving execs and programmers to be held liable.

Having these massive robots on our public streets better at least result in unfettered, transparent public access to all the data informing the thing in the time surrounding the crash.

This shift towards autonomous vehicles is utterly unnecessary. They could be simply improving the safety systems of the vehicles to intervene when the driver is about to cause an accident.

Instead they're doing the opposite - expecting the human to intervene when the robot is about to cause an accident.

I see a lot of detailed analysis in this thread, which is impressive given that literally nobody has read anything more than the half dozen factoids that have been released. At best, some of you are looking at maps of the incident site to figure out potential failure cases.

Knowing that you don't know something is as important as having the facts. To harken back to Cheney, these aren't even unknown unknowns, you're basing opinions on known unknowns.

The article says almost nothing about it other than that the pedestrian was "outside the crosswalk." But the details are what matters. Without the details it is not possible to say if striking a pedestrian was avoidable. There is also a large grey area: Could the collision been mitigated? What other risks were involved?

The bad outcome would be that it turns out that pedestrian safety is too underdeveloped in this system to really be safe.

I have not been following the development of self-driving car tech very closely, but I have some familiarity with the difficulty of the challenges involved and I have a feeling that we are at least two decades away from having fully autonomous tech authorized for use on public roads. Am I underestimating the progress of the tech? I have the impression that there is a tremendous amount of unjustified hype in this field.

I'm really shocked at the trend in comments here.

Yes, we get it. Fatalities will happen sometimes, they are unavoidable sometimes, and what matters in the long run is if we can achieve a significant overall reduction in fatalities.

But my god, a person was just killed by a computer. Can't we have some compassion and humility?

Let's, as a community, set the standard for how we will react to these events. Let's make sure Uber releases detailed data on what happened, whether they were at fault or not. Let's hold the media accountable for their reporting. Let's mourn the loss of life and think about how we can solve these problems.

But for christ's sake, please stop posting the same thing everybody already knows, which gets posted on HN whenever a self-driving article comes out.

Can't we do better?

---

EDIT: I clarified what I meant here [1]

Wasn't trying to say this thread should be all about mourning. This is HN so we should talk about technology, even when a person died. I'm pointing out one specific argument that gets repeated over and over in place of a substantive discussion, and I think we can do better.

[1] https://news.ycombinator.com/item?id=16621589

https://www.abc15.com/news/region-southeast-valley/tempe/tem...

"The fleet of self-driving Volvos arrived in Arizona after they were banned from California roads over safety concerns."

I wonder how software updates and are validated and tested before going to production in autonomous cars. This is scary - imagine being a developer responsible for a similar malfunction. I'm not suggesting that was the case here - just think this might need as much validation as medical equipment (hoping that it won't stifle innovation).

Meanwhile...

https://www.azcentral.com/story/news/local/arizona/2018/03/0...

We changed the URL from https://www.wcpo.com/news/arizona-police-investigating-self-... to one that is longer and doesn't autoplay a video.

Both the computer and the driver failed to avoid the pedestrian. It's possible that the backup driver wasn't paying attention and the technology for autonomous vehicles is not ready. It's also possible that the situation was challenging for both the computer and human and that both failed.

I wonder how many different "cultures" do drivers have?

People from different localities drive differently. I hope they use drivers from say, NY, LA, and various small towns to teach the AI.

People that lives in towns with ~ 10k people drive differently that a new yorker on the same town.

EDIT: They might be training with a biased sample

http://www.autonocast.com/blog/2018/1/18/46-part-two-of-a-co...

I think the test cars should be covered in layers upon layers of foam or another kind of shock absorbing material. This should reduce the collision impact at least a little and can be the difference between someone getting killed and escaping with injuries in some cases.

This comes just few days after this: "When an AI finally kills someone, who will be responsible?" url: https://news.ycombinator.com/item?id=16584862

This seems like a good time to question if we really should have cars at all. Autonomous cars might be better in some ways but they are still tons of metal flying at high rates of speed. Accidents will happen and people and other animals will die on a regular basis. Cars also require tons of infrastructure that walking and biking don't. You also have the environmental destruction of GHG emissions and from the mining of rare metals needed for batteries, you have roads and parking lots destroying habitat and using up valuable space, air pollution and smog from tailpipes or energy generation, noise pollution, and more harmful effects. Autonomous cars might mitigate some of those things but most of it will still be there (or we'll just use cars even more and so we'll have just as many deaths total along with the rest of it?). What about doing research and testing a world without any cars? It will probably be a better one.

"Uber is now the target of at least three potential class action lawsuits, at least five state attorney general investigations, and an inquiry by the FTC because of Sullivan’s decision to pay off hackers and the cover ups."

via twitter.com/adamscrabble

I feel like I should share this here.

The first person killed by a car:

http://www.guinnessworldrecords.com/world-records/first-pers...

There are roughly 3e8 cars in the US, they kill 3e4 people a year => each car kills 1e-4 person a year. If we have say 100 AV right now in US, each car kills 1e-2 person a year. So that's 100 times more dangerous than humans...

If the car is deemed non-responsible it makes it the perfect crime...

How long before someones tries to program a car to "hunt" a victim and hit her when she makes a mistake and it will appear it was a pure accident with no-one at fault?

I once read somewhere that privacy issues (which are very important to us, IT people) were not treated enough because they didn't have enough blood on their hands.

At least with automated cars, there will be blood.

As someone who cannot drive because of a disability, I am putting real hope into self-driving cars becoming a thing during my lifetime. Uber may've just set that back a decade.

And now the costs of developing autonomous cars has been socialized to those unfortunate enough to be walking in front of them while they learn.

Perhaps her estate should receive a perpetual royalty.

Just curious: Do tech companies “pay” the state so they can try out their latest and greatest? Or do they look for states with the least amount of restrictions?

Could this help Uber?

American's seem to enjoy the "X doesn't kill people. If only more people had X, less people would have to die." argument.

So, who is going to jail (or at least - trial) over this?

If you skip the emotional part of the news, the interesting question is: who is responsible when a self driving car made by a company kills someone?

I suppose speculators will speculate - but c'mon. just wait and we'll have video and telemetry and this whole conversation will be moot.

Those who say it's not safe enough: given that no engineered system is 100% safe, when/how can you say that a system is safe enough?

Given that so many of autonomous car accidents logged by the California DMV seem to be the fault of the human driver trying to take control, I’m surprised that the accident here reportedly happened in autonomous mode, which for all its flaws seems to have been good at not hitting things in front of it. In fact, seems like most of the CA accidents were the autonomous car being rear ended or side hit from other cars, usually for going too slow or stopping unexpectedly.

Everyday I see crazy, drunk or angry people just walk across the road recklessly. Some seem to walk in defiance of the cars.

So, what's the statistic of actual self driving cars killing people vs the actual people killing others or themselves.

When will this company just go away ... after everything it's done ... it's now killing innocent people!

I'm paranoid when I'm on a bike. Any road with a dedicated bike lane may be classified as bike-friendly, but I can't consider it so if cars are on the same road. IMO, the only bike-friendly routes are where there are no cars. I wish pavements could be made wide enough for bikes everywhere, so that bikers could be on the road only for a limited amount of time while crossing intersections.

There were an estimated 40,100 motor vehicle deaths last year, or a drop of 1 percent from the prior year.

Just out of curiosity, how many people have been killed by human-driven cars on the same day in Arizona?

Uber's self-driving program seems to be one of the most unethical, most poorly-run in the entire industry. There were plenty of reports before warning us of the questionable quality of their research: cars changing lanes suddenly, blowing red lights, and now one finally killed someone. Every prior incident should have been treated as a serious matter.

Waymo et al has had none of these issues. Time to revoke their licence to test?

Arizona had 10 pedestrian deaths in a single week: https://www.azcentral.com/story/news/local/phoenix-breaking/...

Did the person riding along attempt to take control? I wonder if they will get held responsible..

If only there was this much outrage and discussion every time a human driver killed a pedestrian.

there is a standard in motor industry called misra, because the target user is human you have to met some certain safety level. Otherwise bad things happen like this.

Maybe self driving industry needs to have some safety rules too to prevent when the unexpected happens.

The various reports on this aren't clear- did the car hit a pedestrian, or a cyclist?

I don't see why this lesson could not be learned on a test track or simulator.

Headline should read 'woman crossing street kills electric cars'

Sad to hear it's happened this soon but this was inevitable.

Is this the first death ever?

(Do we count Tesla's autopilot as self-driving?)

Uber is responsible for this, and corporations are people, right? That reminds me I have an episode of opening arguments to listen to on the subject.

There are roughly 3e8 cars in the US, they kill 3e4 people a year => each car kills 1e-4 person

This will be the first major test of liability and insurance with a self-driving vehicle.

Why am I not surprised the first death of a pedestrian came from a Uber driven car.

Ubye

"Uber has paused self-driving operations in Phoenix, Pittsburgh, San Francisco and Toronto, which is a standard move, the company says."

...there's a "standard move" when one of their cars kills someone?

I don't think that really came across the way they wanted it to.

This may not have been an unexpected circumstance. The "pedestrian outside of the crosswalk" report is likely wrong. There is visual evidence that strongly supports that this was a case of the Uber hitting a bicyclist while merging into the right turn lane.

Here's a picture of the crumpled bicycle (note dent on the front right of the car): https://twitter.com/daiwaka/status/975767445859287042

The sign it is lying next to lets us locate the exact point of the collision on Google street view: https://www.google.com/maps/@33.4370667,-111.9430321,3a,75y,...

It's right where the vehicle lane crosses the bike lane approaching the intersection. Very unlikely a pedestrian was walking a bike there. Much more likely the Uber hit the bicycle while merging.

EDIT: I take back "very unlikely" a pedestrian was crossing there. There is in fact a very weird "X" pathway in the median that pedestrians are not supposed to be in. I didn't see that until I looked at the satellite view.[1] So possibly she emerged from there and was walking her bike across the street. That still looks pretty bad for Uber though, since that street is a straight shot with clear visibility and the dent's on the right side of the car. Meaning it would have swerved left to avoid a late-detected obstacle rather than swerve right away from her entering the roadway. She also would have literally had to walk straight into oncoming traffic. I still think bicyclist is the most likely scenario and it's odd the PD doesn't mention the crumpled bicycle.

[1] https://www.google.com/maps/@33.4365931,-111.9425027,198m/da... (street view shows a "no pedestrians" sign)

> You can't really get away from this without putting these cars on rails.

We should just really build rails. Self driving cars are dumping billions of dollars onto a problem better solved by rails. Several cities already have self driving trains including London, Singapore and Kuala Lumpar, and they transport millions of people every day without incident:

http://penguindreams.org/blog/self-driving-cars-will-not-sol...

Self driving car research is asinine. It won't even being to solve the transport problem in America and it's round-peg/square-hole tech. It has the cool factor but there are existing tech that are insanely more useful for a fraction of the cost. American hate public transportation (thanks GM) that we'll never see real/good transport implemented in America.

Calling these pre-programmed multi-MJ wheeled computers "self driving" is foolish and deceptive.

It's likely this was actually a bicyclist, not a pedestrian.

The police are saying pedestrian walking outside the crosswalk but here's a picture of the crumpled bicycle: https://twitter.com/daiwaka/status/975767445859287042

You can see it's next to a sign. That sign lets us determine exactly where it happened on Google street view: https://www.google.com/maps/@33.4370667,-111.9430321,3a,75y,...

It's exactly where the vehicle lane crosses the bike line approaching the intersection. Mostly likely the Uber and bike collided when the Uber was crossing into the right turn lane.

There's no indication that this is the fault of the car or its human driver. Unfortunately, the media won't notice that. She stepped out in front of the car and since she was struck immediately, it sounds like the car had no time to stop or swerve, let alone time for the human driver to react in any way.

EDITed to remove offensive word.

Why am I not surprised Uber is the first company to kill someone. With their top notch ethics, ah. This actually makes me really angry, more than I had anticipated.

Have to quote Prof. Teddy here.

Context: How new technology is introduced on public streets and we have no choice or say on whether we want it or not, even if it endangers us and gets us killed.

> When motor vehicles were introduced they appeared to increase man’s freedom. They took no freedom away from the walking man, no one had to have an automobile if he didn’t want one, and anyone who did choose to buy an automobile could travel much faster and farther than a walking man. But the introduction of motorized transport soon changed society in such a way as to restrict greatly man’s freedom of locomotion. When automobiles became numerous, it became necessary to regulate their use extensively. In a car, especially in densely populated areas, one cannot just go where one likes at one’s own pace one’s movement is governed by the flow of traffic and by various traffic laws.

and

> Even the walker’s freedom is now greatly restricted. In the city he continually has to stop to wait for traffic lights that are designed mainly to serve auto traffic. In the country, motor traffic makes it dangerous and unpleasant to walk along the highway. (Note this important point that we have just illustrated with the case of motorized transport: When a new item of technology is introduced as an option that an individual can accept or not as he chooses, it does not necessarily REMAIN optional. In many cases the new technology changes society in such a way that people eventually find themselves FORCED to use it.)

Source: The Mainfesto: INDUSTRIAL SOCIETY AND ITS FUTURE http://www.washingtonpost.com/wp-srv/national/longterm/unabo...

I’m pretty sure if Arizona’s lawmakers understood the underlying nature of machine learning and autonomous driving, they wouldn’t allow self driving cars to operate.

I don’t care how good your models and data are, until you’re able to write an algorithm that can fully handle and learn from a situation it was completely unprepared for, it will always encounter edge cases like this.

This is what happens when we have a society of people saying AI is already here, people like Elon Musk saying AI is a “significant threat” despite "AI" being essentially a black box statistical model in it's current form. From what I’ve seen, the general public thinks we’re 20+ years of AI than we really are.

You need general AI before AI can drive a car, full stop. Otherwise, you need to isolate roads from other human drivers pedestrians, and cyclists. It just won’t be reasonably safe until then.

The video mentions she was on a bike and shows a mangled bicycle. Also the line markings on the street indicate there is a bike lane to the left of a right turn lane. Definitely very bad.

https://www.google.com/maps/@33.4369934,-111.9429875,3a,75y,...

This is purely an emotional initial reaction to hearing this without knowing most facts, but I just have to vent and for once be part of the outrage culture and make emotionally charged statements before more info comes out.

Uber ruining self driving cars that can save 20k lives each year and improve our economy and get millions out of poverty and improve everyone's quality of life by being short sighted, financial driven, and bringing the worst aspects of Silicon Valley to the extreme.

I am so sorry to everyone at Google and Waymo who have tried so hard to not let anything like this happen by being extremely cautious and safe. Ughhhh I feel so frustrated, this is going to cause so much backlash.

Travis, Anthony and whoever else pushed for this to go ahead without caring about safety is evil. This action doesn't have a direct correlation of evil like directly murdering someone. But these highly intelligent individuals ignored their conscience and the easy deduction that actions to compromise safety, laws, decency for quick profit are immoral. Whether they admit it or not they KNEW that this was a high possibility and refused to take precautions creating this awful culture. 20k lives a year, and all the benifits that will be delayed for years are on their head.

They won't face any consequences, just convince themselves of absolution of guilt and do nothing to remedy the situation, but I have no respect for them.

If human driven cars kill 1000 people a year, and autonomous cars kill 100, will it be politically unacceptable to switch to autonomous? I suspect it will be.

What I don't get is that if this is a test car, how come the the driver did not take over the helm immediately. The driver in that car should be in full alert as if driving the vehicle himself and jump in as needed. Test should not mean to let loose an autonomous car.

Could you please not use allcaps for emphasis in HN comments?

This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.

could there be some way of emitting a signal on your person so cars never get confused? seems like a pain in the ass, but maybe it would be worth it? cars can know for sure there are people wherever the signal is emitted from? dunno if its feasible, curious

[unconstructive but still..] So the day I could be emprisoned because of an error of coding... Like a segfault on an highway... as finally came :D Is the disclamer "no waranty of anykind" shield us from that kind of thing ? ;) (I code on some mainstream C libraries)

Funny how this story has been pushed off the front page of HN prematurely.

I bet people at Volvo are feeling a bit anxious at the moment for that deal:

https://www.media.volvocars.com/global/en-us/media/pressrele...

Uber: 1 Pedestrians: 0 Game on.

On purpose to protect something "more valuable," or on accident?

Not to be the guy who says "I TOLD YOU SO" but it had to happen at some point, sadly & unfortunately - if this is the first (known) case of A.I. controlled cars killing humans, the number of casualties can only grow from now onwards

[flagged]

She stepped out in front of the car in the middle of the road, it's not like a human driver would have done any better.

Interesting nobody used the words jail or murder yet. Not even manslaughter.

Just because there was no driver does not keep it from being a crime. Do the engineers go to jail or do the managers or the investors?

To play Devil's advocate, there were over 100 people killed in the US TODAY from non-autonomous vehicles [1].

Obviously we'd need to know the number of miles driven in each category to get a meaningful comparison, but let's all keep in mind a USA where only one person per day is killed on the roads would be a two order of magnitude improvement over today.

[1] https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...

Given the number of fatal pedestrian hits by human drivers, I'd be interested to see how Uber's self driving cars stack up against the apes.

Having said that, one concern one has with self-driving cars is that one can't "reach an understanding" with them the same way pedestrians and drivers regularly negotiate how they're going to behave with each other.

If one doesn't know one is dealing with a self-driving car, one might consider certain situations safe, because they usually would be, but the car negotiates and notices things differently than a human.

Look, I'm all for taking self-driving cars seriously. I've written about how they're effectively WMDs if the software update computer gets owned. I've also frequently commented about the unethical practices that Uber has taken in the past.

That being said, let's not over-react. This is a single death. These are going to happen. Right now self-driving cars are essentially teenagers learning how to drive. En mass there is going to be a death here or there, but, also en mass, we're going to have a permanently safer road once they've learned, if we can secure them from cyber attack.

Try to avoid all-or-nothing thinking here. Definitely advocate for regulations, and maybe Uber wasn't safe enough, but people are either going to die from self-driving cars or they're going to die from self-driving cars taking too long because of public outrage.

The title here should be amended to “...hits, kills jaywalking pedestrian...”. Let’s get the facts straight. This person was walking outside the crosswalk. The fact that a computer was behind the wheel is irrelevant. A human broke a law designed to prevent this exact scenario, and the result was both predictable and tragic. It’s terrible, but ultimately the person’s fault.

> “They are going to attempt to try to find who was possibly at fault and how we can better be safe, whether it’s pedestrians or whether it’s the vehicle itself.”

Autonomous vehicles should automatically be 100% liable in every accident in which a person is injured, and it should be paid for by a levy on every vehicle.

A levy is fair because the per vehicle risk is much more uniform than with a driver behind the wheel.

100% liability is fair because the software should be written so that the vehicle predicts the path of all objects within its range and adjusts is speed/trajectory so that there is negligible risk of collision given the worst case behaviour of the object. If an object doesn't show on the car's sensors, then its still the car's fault as the sensors should be built to they do pick up all objects. There is a complete power imbalance otherwise: software cannot be injured in a collision but people can, so all people should be 100% covered to give the software incentive to be the best it can.

Summary: a car is governed by predictable physics, hence there is no such thing as an accident involving an autonomous vehicle, only mistakes.

The problem is likely a failed ABS Brakes and on all cars. When the brake is applied and then it fails, there is less than 5 seconds to a crash and likely fatality. No time to react. Experiencing the brake failure on two cars, I have tried to get the attention of NHTSA for a decade. NHTSA has a lock on faulty research to blame an innocent driver for all accidents instead of finding the correct root cause of the crash. My estimate is 50% to 70% of all fatal crashes is caused by a defective ABS Brake.. Finally with no driver to blame, NHTSA will have to solve the correct fail mode for these crashes. The Congressional Sudden Acceleration investigation found nothing to make the roads safer and Toyotas have SA crashes today and the certified accident investigator only logs items to blame the innocent driver. Unless the braking technology is totally changed in self driving cars, I expect daily crashes with no one to blame. I have sent Elon Musk letters that the Tesla has ABS Brake issues causing crashes, but their engineers join in blaming the driver and do not fix the problem Accident investigators are clueless Lawyers and DAs are clueless News Reporters are clueless NHTSA is clueless NTSB is clueless Carnage on the roads goes on unabated and we convict an innocent driver.

On the bright side, with an autonomous vehicle, an accident is a learning experience that will benefit other autonomous vehicles.

Striking a pedestrian in a crosswalk is horrible and terrible news. The reality though is how many pedestrians are hit by manual driver's vs automous drivers? On a percentage basis I gotta believe autonomous cars are orders of magnitude safer. Self driving cars aren't going to be perfect. How many people lost their lives in early factories to machines during the industral revolution? Imagine if they had pulled the plugs back then.

Reply on news.ycombinator.com