Hacker News Reader

Comment by vichu | Hacker News Reader

This is potentially precedent as a legal blocker for public adoption of self-driving cars. If I am to serve a 1 year prison sentence for faulty code or a deprecated LIDAR sensor, I don't see in what scenario I would be willing to leave my fate in the hands of a self-driving car manufacturer. Just as in the case of a chauffeur, if they are driving and commit vehicular manslaughter, I would not expect to be deemed guilty.

It seems to me that it would be the benefit of self-driving car companies to own up to liability as it is to their benefit of achieving widespread adoption. For example, Volvo is in line with this idea and has publicly stated that they would accept full liability in fully autonomous operation modes[0].

In this case, I do think that some liability lies with the driver - as they were tasked specifically with prevent situations like this. What is not clear is whether or not this task is even humanly feasible given reaction times, and based off of this, whether or not Uber has been criminally negligible. Given this, I am surprised that the prosecutor seems to have absolved Uber of any blame.

[0] https://www.media.volvocars.com/global/en-gb/media/pressrele...

View on HN

I'd really like to see self-driving car liability treated the same way airline crashes are. Less "whose fault is this" and more "how can we prevent this same accident from ever happening again?" The second is the really important question and the one we need to keep answering to reduce the toll of driving deaths. This will mean the cars driving on our roads getting more and more regulated over time but I'm quite OK with that.

Airline crashes absolutely are treated as "whose fault is this". That's why airline crashes usually have lots of lawsuits, with millions of dollars of damages, filed after they happen.

Every NTSB report has a "probable cause" section determining who or what was at fault. And then the FAA hands out fines and courts award additional damages largely based on that finding.

It is true that the punishments aren't handled by the NTSB but that's more about separation of concerns & departmental remits than anything else. Is separating out "fault finding" from "punishments" a good idea? Maybe? I honestly don't know.

> Every NTSB report has a "probable cause" section determining who or what was at fault. And then the FAA hands out fines and courts award additional damages largely based on that finding.

In case everyone is not aware, NTSB reports cannot be used in court to establish fault or otherwise be used as evidence when seeking compensation for damages.

"No part of a report of the Board, related to an accident or an investigation of an accident, may be admitted into evidence or used in a civil action for damages resulting from a matter mentioned in the report." 49 USC 1154(b)

I have a question..

In criminal cases what happens with inadmissible evidence? Like fruit of poisonous tree doctrine... what if you literally have video proof and tons of other graphic proof of a murder, you really just ignore it and let the perp go back into society?

if you have tons of proof about a murder, it's almost certain to fall under one of the exceptions to the doctrine. e.g, if law enforcement would have discovered the evidence anyway, it is not excluded.

In principle, it’s inadmissible. So, theoretically, yes. The perp just goes free. Not very common for a lot of reasons.

There are exceptions to the doctrine for exactly the reasons you mention. The legal system isn't as insane as people like to make it out to be.

The part you missed is probably the post mortem investigations are why air travel is the safest form of transportation.

> "how can we prevent this same accident from ever happening again?"

It would be nice if society adopted this with all crimes. It's not like we can prove we're any different than the low level design of a machine; when it comes to the "free will" ideology currently the justice system is designed around.

This. For me believing in free will is just another type of religion. It makes us feel safer and that we're in control. I think is useful for that, but we're deluding ourselves.

And I just realized you're being downvoted. How surprising.

Even if free will is an illusion it my be maladaptive to design a justice system without consequences for perceived faults.

It may be the downfall of humanity progress as well and things may change.

I am not advocating a 'justice' system without consequences. You will have consequences. Consequences are just another input into our brains that do help deter some people from some behaviours. I don't kill people because I'd go to jail. I don't kill people because I don't want to hurt them (and that is on my programming, upbringing, possibly nature to a degree as well). I think the only way I'd kill someone is by accident or in a spur of the moment when emotionally disturbed by an incident (me or my family fear for our lives). Society has mostly accounted for those cases and I may not even go to jail if I did it, because people largely can relate to those situations and they agree it's not 'fair' send me to jail for them. However, there's several sectors of the population no one seems to be able or want to relate to (poor people that got little opportunities and guidance in life, abused people, mentally ill, psychopaths, a mix of all of that etc), and the people creating the rules therefore decide to 'punish' them in the name of justice for being bad and because we have not devised (and not invested enough time either) a better and more humane way to deal with them. In theory we want them to be reformed, in practice, is mostly about retribution and putting undesirables away.

We just can't help downvoting him.

Since everyone was socially conditioned to desire and believe in free will. Similar to the past with God but I would have been hanged. It will change and it’s her not him.

Of course it’s a religion. People are deluded to think otherwise.

Well if you stop viewing it as fault and instead view it as incentives it makes sense. If you (intentionally) murder somebody, you still need to be punished (even if technically "you" are not at "fault" because it's not "you" it's just the algorithm running in your brain) so that others will be disincentivised to do the same thing in the future (i.e. the algorithm running in most people's brains will take the potential punishment into account when calculating whether the crime makes sense or not).

To a large extent we already do that, e.g. many countries have upper limit in punishment (like 30 years jail max), or allow criminals to delete their criminal past after some time has passed, not because that "just" or "fair" or whatever, but simply because it's "effective".

I agree with you, and I think you also agree is not quite effective. It's a factor in deterrence, but we need to do more to understand people that still commits crimes and solve more of the root causes. We won't achieve perfection probably but we can do a lot better IMO.

I think we need to educate people how free will is an illusion and because it would help in preventing the root causes.

Whether or not free will exists (I'm a compatiblist) there are certain cases where the threat of punishment will cause people to change how they act and there are certain cases where it won't. If it were the case the the people at Uber intended to kill someone then the threat of punishment might have prevented that and Elain Herzberg would still be alive. But as the people at Uber did not expect that their actions would result in her death the threat of punishment isn't going to be very effective at preventing deaths like hers.

The distinctions our legal system makes regarding coercion, mens rea, insanity, etc are mostly trying to get at this same underlying distinction.

Society by and large does do this, just not with a grandiose show at every "incident".

"Whoa, there sure are a lot of fights at bars we have to keep responding to! Maybe those would happen less if we had fines for bartenders that had too many fights? Or maybe if they had to get a license that we could revoke if they weren't preventing fights?"

"Whoa, it seems that a lot of people don't own guns, but then go and buy one when they get pissed and shoot somebody. Maybe that would happen less if we had waiting periods?"

I do like this idea, where legal action is more about preventing the event reoccuring rather than revenge or payback. Unfortunately "an eye for an eye" is a very strong instinct and will take a lot of education to allow society as a whole to move beyond.

>This is potentially precedent as a legal blocker for public adoption of self-driving cars.

I don't think it is because in this case the person was hired by Uber specifically to "drive" this car. They knew that the car is still an incomplete product, they knew that they were testing it, they knew that they were literally getting paid to keep this car from doing something like this.

That's very different than a random person using a self-driving car that they expect and were told is a finished product.

> I don't think it is because in this case the person was hired by Uber specifically to "drive" this car. They knew that the car is still an incomplete product, they knew that they were testing it, they knew that they were literally getting paid to keep this car from doing something like this.

Sadly, while that is in fact the task of the safety drivers in other automated car trials it wasn't the case here for uber, because the "safety driver" was given both the job of safety driving and of checking, controlling and classifying the automated system's information, ensuring they'd have to split their focus and would need to frequently context-switch between two completely different tasks.

Worse, this decision of Uber all but ensured this would happen: the automation system is more likely to lose its shit and need checking specifically in situations where the safety driver would be most useful.

Don't get me wrong, Uber isn't fault free here in my opinion, and I'm mostly in aggreement with you. Pretty much everyone involved here was negligent in some way if you ask me.

My point was only that I don't think this would set any kind of precedent since it's so far from a simple case of "an owner of a self driving car".

> What is not clear is whether or not this task is even humanly feasible

Not watching TV while you’re supposed to be driving is a pretty low bar.

But apparently, humans can’t even be relied on to do this, even when it’s their one single job.

If you think it’s easy, go sit in the passenger seat without talking or using your phone and see how long you can maintain focus.

There are decades of experiments showing that boredom and fatigue are huge problems for this kind of work where someone is mostly idle except for rare events. It’s why the TSA mixes fake images into the stream on X-ray scans so people see things regularly and don’t zone out. Anyone smart asking humans to do repetitive work builds layers of safeguards in to keep people engaged – or in Uber’s case they just hope they can shift blame to the person in a bad design, which worked so far.

Uber should have had people remote-monitoring the monitors at this early stage.

I saw a different company’s car today: they’re being responsible and have two people in the car.

Pretty much all of them do. In fact Uber used to do that before they moved to AZ. Having a single person responsible for driving & monitoring was a way to save cost and log more miles.

Also >"The system is not designed to alert the operator," the report notes.

Aren't driving instructors supposed to do pretty much that? Supervise an imperfect driver and take control quickly if the situation requires it?

Driving Instructors are always actively engaged, atleast mine was, he'd always check traffic, plan the route, give me directions, etc.

It's way more active than watching a self-driving car doing it's thing completely autonomously.

I think the digression to whether the concept of a safety driver is flawed from the start irrelevant. Sure, maybe the process of having a safety driver backup is physically inhumanly possible because they'll get road blind and won't be able to react in time. But that didn't happen here. We have no evidence of that ever happening in self driving tests. Of the millions of self driving car test miles with safety drivers not once has a safety driver been looking at the road, but not able to react in time because of the monotony of the job.

What we have here is someone who was watching TV on the job. I adamantly refuse to believe that humans are incapable of not watching TV for a stretch of several hours.

If the driver had nodded off, looked out with glazed over eyes, or any number of other situations consistent with your scenario, then you'd have a point. But this was not that case.

> I think the digression to whether the concept of a safety driver is flawed from the start irrelevant. … What we have here is someone who was watching TV on the job

The fact that this particular person dealt with boredom by watching a video makes it easier to blame but that's just one of many ways in which people cope with boredom and it's not the root cause. You can blame the worker if it makes you feel better about yourself but if your goal is to reduce the number of errors the system has to be redesigned not to depend on people acting like robots rather than humans.

> Of the millions of self driving car test miles with safety drivers not once has a safety driver been looking at the road, but not able to react in time because of the monotony of the job.

Do you have any evidence supporting this claim? In particular, you'd need to prove that all of the self-driving car tests have the same one-person setup (which we know to be incorrect), every company ignored decades of well-understood risks and similarly didn't have any tasks for that person to perform and thus stay engaged, and you'd need to know how frequently incidents occur which require driver action to prevent a problem.

It's far more likely to be the case that there have been many situations where someone was distracted or focused on an area other than where a potential risk was but the other driver or the self-driving system successfully avoided it turning into an incident which made the news.

This person’s job would be more accurately described as “fall guy” than “driver”. Maintaining constant attention toward a system that works properly by itself for hours, with a response time of seconds to correct its mistakes, is not a reasonable task for humans. This guy’s role is to take a paycheck for doing virtually no work, at the exchange of taking the risk of being blamed if things go wrong.

I know it’s glib, but just on the face of it, I find the idea of Uber taking responsibility for anything a laughable proposition.

> If I am to serve a 1 year prison sentence for faulty code or a deprecated LIDAR sensor, I don't see in what scenario I would be willing to leave my fate in the hands of a self-driving car manufacturer.

What if we came up with a reasonably objective test of driver skill and the car outperforms you?

This is and remains the standard argument - the car doesn't have to be perfect, it just has to beat a human having a bad day. If you are worried about the liability, maybe keep the car serviced and sit in the drivers seat and look out of the window instead of watching a movie. Tell the car to dive slowly maybe.

I feel sorry for the driver who is creating this precedent, but this is hardly a road block. The average human is not a great driver.

> If I am to serve a 1 year prison sentence for faulty code

I wonder why would a programmer go and work on a code when he is always in danger of going to prison for every mistake he makes. We all know that bugs in software happen and it’s impossible to write bug free code

> I wonder why would a programmer go and work on a code when he is always in danger of going to prison for every mistake he makes.

You don't go to prison for "mistakes", you go to prison for criminal negligence.

Given that you can go to prison for criminal negligence in any other profession (e.g. a dentist who exposes thousands of people to HIV), why would programmers be the one category of job that's exempt from criminal laws?

> You don't go to prison for "mistakes", you go to prison for criminal negligence.

The difference between the two is what a politically-motivated prosecutor can convince a jury of.

If a politically motivated prosecutor can convince a jury of twelve people that Bill killed Dave, by using his psychic brain powers, or, alternatively, by sticking pins into a voodoo doll, or some other form of non-scientific witchcraft, Bill is going to hang.

The hypothetical gullibility of juries doesn't mean that we shouldn't have laws against murder, juries, or prosecutors.

The line between mistake and negligence is a lot blurrier than the line between killed and didn’t kill.

If the programmer can’tbe bothered to put in enough verification effort to avoid being negligent, they should not be in charge of safety critical code. Plainly: don’t pretend there aren’t ways to ensure correctness of systems.

But that same developer can put the same effort in at another field and not be constantly at risk of a mistake doing massive damage to them. You would have to be paid an incredible amount to make it worth it.

Why is that a problem? The cost of business should obviously include safety concerns.

It's "not a problem" until the next round of "why does Silicon Valley only work on trivialities?" articles pops up.

Or there will be people who gain expertise in developing software good enough that software engineering will be considered an actual engineering discipline.

How do other fields where the workers are liable for damages like this work? What do architects who sign off on buildings do for example?

I don't believe they all ask for incredible amounts of compensation

I think software is a lot more brittle than physics. It takes a lot to go from a sturdy building to something that could collapse, whereas in software the difference could be a single line of code, or two transposed words.

Also, the software world doesn't have the benefit of millennia of accumulated best practices.

Finally, the senior engineers and architects who are licensed to sign off on things they will be held criminally liable for do get incredible amounts of compensation as compared to their more junior colleagues.

Anybody can build a bridge that doesn't collapse, it takes an engineer to build bridges that barely don't collapse. This is hard work, don't downplay it.

I'm not sure how facetious you're being so I'll just play it straight. Modern buildings are not supposed to barely not collapse, they're supposed to be safe even under scenarios quite a bit more extreme than what they're expected (or even legally allowed) to handle.

They're supposed to be exactly as stable as required by law for their intended use, using the minimal amount of labor and materials so satisfy the requirements. Making them more stable using more money is easy, but getting the calculations just right requires years of training.

I see what you mean now. My point is that even if there's a minor screwup, the architect probably won't be prosecuted for anything because the building won't fail catastrophically thanks to modern standards which have substantial margins of safety built into them. It really does require criminal levels of negligence or extreme circumstances that would almost certainly save the architect from prosecution to have a building collapse on you.

On the other hand, a minor screwup in software is far more likely to cause catastrophic failure because we just don't know how to workably build large, robust systems out of code.

Point well made.

Nit: meeting the requirements of law isn't barely collapsing; that requirement has so many safety factors built-in because the building code has to approximate so much. The approach isn't that dissimilar from how that "anybody" you mention would build their bridge that doesn't fall down: by guessing safely.

It's over 20 years since I finished my degree, and I've never worked in the industry, but I spent sufficient hours poring over the ISO standard document for pressure vessels as part of my final year project that I can still remember the sorts of things it covers.

Effectively, they test things to destruction, then publish minimum requirements. So if you want to pressurise your reactor vessel to reactor vessel to 30 atmospheres, you can pretty much look up a table that'll tell you precisely how thick the reactor walls need to be for each of the commonly used materials. If you want to use something uncommon, then you need to pay somebody to test it.

If it fails in a catastrophic fashion, you can expect to be asked to show that you did your due diligence, and there are extenuating circumstances a reasonable engineer could not have been expected to foresee and plan for. Or that you did foresee it, and somebody else chose to accept the (clearly defined) risk.

There are non-tech professions, where the lives of other people are literally in your hands. And a lot of the people doing them get paid a lot less then software engineers.

> We all know that bugs in software happen and it’s impossible to write bug free code

The go-to counterexample is the code that ran on the space shuttle, which took years and hundreds of people and $200 million to produce. I have been told that nobody has ever found a bug in the final production system. The development practices involved in creating it seem like they would make most software engineers want to curl up and die. One tidbit from the following link I found is that 60 people spent years working on the backup flight control system, intended to never be used!

https://history.nasa.gov/computers/Ch4-5.html

Well we could mandate that self-driving cars be programmed to the same standard. I mean it means that self-driving cars will simply never exist, but we could mandate it!

Maybe we shouldnt run safety critical code of which we dont know what it does. Hard realtimesystems exist for a reason. Sure it would also be nice if we could drive nuclear power plants at home via a smartphone app, so the most qualified expert could help in an emergency, but its simply not a sane idea.

This cop out of "Software gonna have bugs" as a way to evade all liability doesnt hold in any other profession. I dont see why we get special treatment here.

It does hold, in some form or other, in other professions. The EPA formally defines how much cash they're willing to burn to save a human life, which seen from one perspective is an "environment gonna kill people" cop out. Nuclear missile silos have two operators because "people gonna launch nukes". Cars have airbags because "vehicles gonna crash". Every field has risks and risk management, every field has certain steps that they could take for the purpose of safety that are judged too expensive to justify the risk, and part of managing risks in software is that you should plan for bugs and plan to mitigate their impact.

Nobody calls the emergency services out because they assume cars are going to crash and plan accordingly, so why does the software engineering industry get called out for assuming software will go wrong and planning accordingly?

Architects and engineers seem to go to work everyday knowing that if they are criminally negligent they could face prison.

My apologies for potentially hijacking this, but... this is exactly why the term "software engineer" bothers me. Yes, the software you write isn't likely to cause a shopping mall to collapse on a crowd of people, but there can be huge financial and societal responsibility here, and yet, almost every software license in existence completely disclaims that. Engineering comes with a tremendous amount of ethical and legal responsibility.

But innovation!

Fail fast!

...sorry for the sarcasm but this is a message student engineers internalize in part because it is pushed by companies they want to work for but can't explain why. They don't have strong boundaries between why it might he a justifiable philosophy for Facebook but not for Boeing.

Agreed. I'm a software developer who has a degree in mechanical engineering and changed over. I cringe at calling myself an engineer in my current role.

First - architects shouldn't be lumped in with engineers in this context. It seems like there's some confusion around how far the responsibilities of an architect extend, which makes me wonder if things are different in the US? Whenever I research it seems to be the same as Australia though. Anyway, architects know as much about structural engineering as engineers know about Le Corbusier. An architect would really have to mess up to be criminally negligent.

But in response to your comment, engineers have a very clearly defined set of rules, collectively called "the code". As long as they design to them you'll be free of criminal negligence (at least during the design & development phase, things get a little murkier in construction).

More to the point, engineering for the real-world means layer upon layer of uncertainty. e.g. civil/structural engineers use materials we can't model (we use pretty-good approximations for concrete behavior) in conditions which are unknown (soil) to resist forces we can't predict (weather, earthquakes, dynamic response to loading). How does the code deal with all this uncertainty? Slap safety-factor after safety-factor onto everything. Whoever comes up with a method for more effectively dealing with this stuff will make millions.

The most obvious example being that we design structures to resist a 1-in-100 year storm. In other words, we expect a structure to be within spitting distance of failure every hundred years. But as long as you design to that standard, you're fine.

The code wasn't faulty. The configuration was explicitly set to not brake, as far as I understand. If someone got in trouble here, it wouldn't be for programming.

Operators of vehicles need to be held personally responsible for the damages caused by putting their vehicles on the road, whether the vehicle is autonomous or not. Additionally there must be a responsibility for manufacturers to not create faulty products, but that can not absolve the "operator" of the product. It should not be said that the "operator" of a Volvo autonomous vehicle is not responsible for the actions of the vehicle under her control, simply because Volvo says they accept full liability. I think your chauffeur analogy is flawed. If I set cruise control in my car to 75mph in a 25mph school zone, am I not liable? It is not possible to delegate our personal responsibilities to machines, and wash our hands of it.

Volvo's acceptance of liability seems little more than a smart business decision- the cost of an insurance payout to the family of the deceased is less than the money they stand to earn in profit from selling autonomous vehicles. We can't allow this generous actuarial calculation to absolve the operator of such a vehicle of responsibility too. If you don't like it, take the train.

Why?

The individual operator (the driver, or the person in the operator seat who presses the "engage autonomous mode" on the car) is the one who chose to buy a vehicle and put it on the road, potentially endangering other people. We need to be responsible for our own actions and the situations we create.

Is their a gun shop owner liable for murder (and/or wrongful death) every time someone is killed with a firearm? What about the clerks at the gun store? Or the truck driver who delivered the firearm to the store? All of their respective actions clearly created a situation where people have died; your position is silly.

You are assigning blame further up the chain, which I am not doing. I stated in my first post that I do believe the manufacturer is liable for their faulty products, but can not be SOLELY responsible. I might find the gun shop owner for selling a faulty or damaged gun, for example if there was a recall and they ignored it.

If you bought a faulty gun with a broken safety catch, and waved it around in a crowded street, and it fired, killing someone, you should be held personally liable. You brought the gun out in public and created the scenario which caused someone to be hurt by it.

What if the ammunition was faulty, and a case of it blew up in someones car killing a pedestrian next to it? Or a gas leak in the fuel line that dripped onto a faulty underground electric cable, igniting and causing injury?

I don't know what the threshold is in other countries, but Canada has a vague but useful threshold: "knew or ought to have known". A different way it's been stated in the courts is "what a reasonably prudent person ought to have known"

Looking at the fuel line question through that lens: did the leak start while you were driving and you were completely unaware? Or has it been leaking for a while and you just haven't gotten around to fixing it.

With the ammunition, how was it stored? Was it dumped in a toolbox filled with pointy screws? (A reasonably prudent person ought to know that ammunition is fired by striking the primer with a sharp object). Was it stored in the front seat of the car on the hottest day of summer? (A reasonably prudent person would expect it to get really hot in there)

Etc. Etc. The thing about negligence is that there's a lot of room for interpretation. As another example, if you've been driving a car around with self-driving features, and you've experienced it behaving erratically multiple times, and that's followed by an accident... you knew or ought to have known that it was dangerous to be on the road in that vehicle. If there was an OTA update for the autopilot last night that installed silently, and it results in a crash, then it's probably the manufacturer who's liable.

Imagine that your only recourse is to go after some big faceless corporation whose immensely complex algorithm failed in some highly unusual way, and no one can really say what happened, not even the engineers who wrote it (and some of whom might have long left the company...)

It's similar to the situation with companies like Google blocking people who have no easy way to get support, except more morbid.

I'm not fond of living in such a world where personal responsibility is dissolved and becomes meaningless.

So if a cab gets in an accident, then the person who hail'd the cab should be liable? What if it is an autonomous cab? If an automated elevator fails and kills someone, is the person that pushed the call button liable?

This is something that I really don't like about our society, the fact that someone "must" take be responsible. When someone goes on a shooting spree, of course they should go to jail. But if they die in the process of committing their crime, there is almost always an outcry for the next person in line to be responsible.

We have rules about elevators. There are elevator inspectors who regularly certify elevator safety, and they are liable if they falsely attest as to its good working order. The owner of the building is also responsible for maintaining their own elevator. There isn't anything the passenger of an elevator can do to meaningfully affect its operation or safety. I suppose if you knew there was someone working in the elevator shaft and you pushed the button to hurt them, you would be responsible for that.

We have rules about motor vehicle operation. The operator is responsible. If you are the passenger in the back seat of a cab, the driver is responsible, not the passenger. If there is nobody else and you are the pilot of an autonomous vehicle, you have to assume responsibility for your vehicle and its actions.

/Or/ we take the way less insane course of action and regulate AV safety, like we do elevators and airplanes?

Reply on news.ycombinator.com