Hacker News Reader

Comment by abduhl | Hacker News Reader

"As you are undoubtedly aware, there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose."

The government appears well aware of this hand waving garbage that companies are trying with "it isn't really autopilot, keep your hands on the wheel wink." This is exactly government's job and is a fabulous example of a government reacting to ongoing events that have proven that so-called autopilot systems are inherently a danger to the public at this stage.

View on HN

Yes. There are a number of federal agencies that are a shit-show at best but I can't remember a time in my adult life when I was disappointed in NHTSA.

The NTSB also does amazing work, though it's the smallest Federal agency (~400 people), which I think probably helps greatly.

NTSB has awesome investigators. Some of the smartest people I've met and worked with in my life. They are very serious and dedicated, while having to put up with a lot of roadblocks, nonsense, and other issues from both corporate America and other government agencies.

Can confirm at least though many years ago, the internal NTSB IT staff/programmers/DB were a joke. I guess that's just government though and like much of government, they relied a lot on outside contractors. For years they were a ColdFusion shop, so I'll let your mind start there.

>inherently a danger

Can you expand on how they are an inherent danger, rather than a possible danger? I only challenge you because Tesla has published several occasions where Autopilot has possibly saved lives by auto-braking.

> Can you expand on how they are an inherent danger, rather than a possible danger?

I do not believe those words (inherent & possible) are in any way mutually exclusive. They might even mean the same thing when describing a latent danger. So I do not quite understand the basis of your challenge. Also, I'm not parent poster.

My understanding of inherent might not be right, then. I think of it as "it is definitely a danger, and thus can only be described as dangerous," which I would argue is not mutually exclusive with possible but is mutually exclusive with "not always dangerous and maybe even anti-dangerous."

So what I mean is, the autopilot could turn out to be safer than manual, in my mind, so how is it "inherently" dangerous.

A swimming pool in your backyard is an inherent danger. That doesn't mean anyone will be injured or killed, but a deep body of water with sheer walls is inherently dangerous.

It would nonetheless also be accurate to describe it as a possible danger, because the danger only manifests in certain circumstances.

> So what I mean is, the autopilot could turn out to be safer than manual, in my mind, so how is it "inherently" dangerous.

That could well be the case but I don't think it eliminates the inherent danger. Maybe it's less dangerous, but cars on the road are just kind of dangerous in general.

I think it's also very generous to assume that the early auto pilot car systems we have today are necessarily safer than human drivers. Emergency braking is almost certainly a net gain. Self driving might be or might not be. A self driving add-on cooked up by geohot? I'm doubtful it's yet gone through the kind of engineering rigor we'd expect of a system like this.

It's inherently dangerous because it's a bunch of hardware and software that take over a 2 ton vehicle to turn it into a software guided missile. That's an inherently dangerous set-up that will need a lot of thinking to keep it safe for general use.

Just like a piano suspended from a rope is inherently dangerous. The danger may not manifest at all for that to be the case.

"So what I mean is, the autopilot could turn out to be safer than manual, in my mind, so how is it "inherently" dangerous."

In the same way that the ocean is still inherently dangerous even if you're wearing a life jacket, or how airplanes are still inherently dangerous even though aircraft autopilot systems are effectively mainstream at this point.

My statement was that "ongoing events that have proven that so-called autopilot systems are inherently a danger to the public at this stage". The important part to note is "at this stage" where at this stage refers to this stage of development and implementation.

Contrary to Tesla's publishing several occasions where Autopilot has possibly saved lives, there have been concrete incidents where people have died while Autopilot is being used. Tesla has backed away from these incidents with the same hand waving that NHTSA has specifically touched on here: that the user is exceeding the system's design purpose.

In this context ("at this stage"), where consumers have historically and will most likely continue to exceed these systems' design purposes and operate them outside of their intended scope, it seems very clear to me that these systems are inherently dangerous to the public. That is, there is no way to make these systems not a danger to the public in their current form.

The NHTSA are fine with auto-braking - indeed, they want it to be standard on all cars in the future[1]. What they're not OK with is companies marketing glorified cruise control as a self-driving feature. Elon Musk likes to lump his implementation of both under the same "Autopilot" umbrella for PR reasons, but you can have the former without the latter - indeed, I'm pretty sure the most prominent example he gave of the former saving a life was wityh the latter switched off.

[1] http://www.autoblog.com/2016/03/17/nhtsa-iihs-20-automakers-...

Proof that so-called autopilot systems are a danger to the public would be evidence that they perform worse than the average driver. I haven't seen such evidence, what are you referencing?

It's the other way around - you need a proof that they perform comparable to an average driver during expected use, and they do not have absurdly stupid corner cases during actually expected use.

Well, humans have absurdly stupid corner cases during actual use, too. It depends on how small the corner is, I think.

Yes and humans require certification before they are permitted to drive vehicles on public roadways.

To get that certification, do they test you first, or do they wait for you to screw up and then take it away from you??

We know, being humans, and that we have the ability to process lots of complex information in a way that's very difficult for computers to replicate. Hard AI doesn't actually exist (yet). We also have a 100 years of humans driving cars worldwide so we understand well what they're good at and what they're not, so laws & safety designs take all of this into consideration.

Each computer system will be encountering new, diverse things in the real world without a good understanding of how they'll perform. There are lots of crazy hard problems here that no one has solved yet. So to suggest we just automatically trust it because humans make mistakes is foolish when the consequences are so high. If someone came out with a surgery "autopilot" tomorrow, would you suggest it start giving triple bypass surgeries right away without FDA approval because humans make errors too?

One of the features of the common human firmware is self-preservation instinct. It lets us trust that our fellow drivers, while still prone to mistakes, won't generally make obviously suicidal errors. Can one say the same about a new ML algorithm running on some board designed half a decade ago? How exactly would one know, without a thorough audit?

They test you first, but if despite passing the test you screw up sufficiently badly later they decertify you.

It's a belt-and-suspenders system.

We've been dealing with those corner cases for thousands of years and we know them pretty well. Given that we all run on pretty much the same hardware and firmware (with minor differences), you could say humans have been thoroughly tested for few millennia, including a hundred or so years of road testing.

So yeah, I a bit of thorough testing of a completely new hardware running completely new software isn't too much to ask for.

so we should wait till a statistically-significant percentage of people die before we tell everyone, "yep, definitely dangerous. Lets add some regulation" ?

I wonder if they could shut him down on the basis of car insurance requirements and the legal definition of "driver". If "driver" is defined as the "person" inputting control commands, then either you dont consider the computer a person, and then nobody is legally "driving" the car, or else the software somehow meets the definition of a "person" or driver, in which case its an unlicensed driver operating the vehicle and insurance may not have to cover it.

The NHTSA has been a strong proponent of self-driving technology and has gone out of its way to make sure there is a path to doing it legally, without pulling the legal 'gotcha' that you described here.

The device would probably be SAE Level 2, and should be regulated as such.

When you measure the safety of autonomous cars by counting the miles they travel, and not counting the miles that they intentionally avoid or defer to a human, you have succeeded in creating a metric that looks like it's useful for comparison, but is actually completely meaningless. It's like measuring typing speed while ignoring typos.

Until autonomous vehicles are subject to the full spectrum of conditions that human drivers face, incidents/mile is not a meaningful measure of comparative safety. And until then, the burden of proof should be on the creators.

Not true. Lots of accidents happen in "good" conditions because humans suck at driving.

The way you compare, is you give a self driving car to a person, and compare actually usage for that person, and see if owning a self driving car increases or reduces accidents for that person.

Being perfect only half the time is still revolutionary.

I have a 100% perfect safety record over probably hundreds of thousands of miles of driving a class B truck with my knees instead of my hands, using cruise control, on straight empty highways in Nevada, Idaho, and Utah, with perfect weather and good road conditions. There's nothing revolutionary about my ability to perfectly drive with my knees...I just very carefully selected the conditions where I was willing to do it.

You may have that perfect safety record, but other people don't.

People still crash every day in good conditions. That's thousands of lives that could be saved with our imperfect, sunny weather on the highway only self driving car.

You underestimate how bad human drivers are even in perfect conditions.

You are completely missing the point, so I'll try explaining via a Reductio ad absurdum hypothetical.

Lets say that Semi-Autonomous Cars are currently tested on about 80% of the tasks that humans currently face in the real world, and that through some feat of engineering and design we don't have to worry about the ridiculously messy transitions between Autonomous Mode and Human Mode.

And let's say that for those 80% of tasks, the Semi-Autonomous Cars have a 0.05% accidents/100k miles incident rate.

And let's also say that Humans on average have a 1% accidents/100k miles incident rate.

What you're telling me is that the Semi-Autonomous car has a much better safety record, so give us Semi-Autonomous RIGHT NOW OR WE'RE ALL GONNA DIE!!!

But what I'm telling you is that you don't know enough to make that decision yet, because you don't know exactly how well humans do on the 80% that Semi-Autonomous cars currently handle, you merely know the average accident rate over the current 100% of scenarios. For all you know, Human drivers could have their average 1% accident rate as a result of a 0.0% accident rate for that 80% subset and a 5% accident rate on the remaining 20% that Semi-Autonomous cars can't handle. And if that were the case, then forcing us all to use Semi-Autonomous cars would actually increase the average accident rate from 1% to 1.004%.

Until you fully understand what Semi-Autonomous Cars are capable of, AND know how well human drivers do on that restricted subset, you can't definitively say that current technology is better than humans.

I understand what you are saying, I am just disagree with the facts.

"Human drivers could have their average 1% accident rate as a result of a 0.0% accident rate for that 80% subset "

No they couldn't, because they don't.

I am asserting that for this specific 80% of perfect conditions, humans are still terrible drivers. And that being better than them is EASY, because of just how terrible humans are at driving (even in "perfect conditions").

> No they couldn't, because they don't.

The numbers were deliberately exaggerated to make the point. The fact stands though that until you know what the numbers are, the best answer is not easy to come by.

> I am asserting that for this specific 80% of perfect conditions, humans are still terrible drivers. And that being better than them is EASY, because of just how terrible humans are at driving (even in "perfect conditions").

And I am asserting the opposite: that the appearance of safety of autonomous vehicles is the result of highly selective conditions with near laboratory levels of control, the likes of which are so monumentally easy to handle that even humans, as shitty and inattentive as they are at driving, can handle with comparative levels of safety. And I certainly think it's possible that computers will lag behind humans for another 20-50 years while we slowly develop the massive body of fast-heuristics research necessary to make NP-complete planning decisions with the speed and capability of even below-average humans.

The frequency with which an autonomous driving system relinquishes control is one important metric for estimating the actual risk. Measures like this are better and faster than using the actual tally of catastrophic events, because there is more, and more frequent, information. Engineers working in high-risk fields have been doing this sort of analysis for decades, and there is no reason to make an exception for autonomous road vehicles.

>>humans suck at driving

Proof?

Well, roughly every driving week or so...

Just two days ago, driving on a roundabout some lady enters the roundabout right in front of my car. Fortunately it was a two lane roundabout and the left lane was empty so I could swerve to avoid crashing into her left hand side door (I was coming straight at her).

At the traffic light I asked her if she had not seen me and she went 'Seen what? Where?', so apparently she had not seen me at all, which is pretty impressive given that I was less than 5 meters away from her when it happened.

This sort of thing happens with some regularity. All it takes for a situation like that to turn into an accident with possibly a fatality is for me also not to pay attention.

beyond the obviousness of what you have overlooked, its also worth pointing out that not all autonomous car usage will be substitutive. What if i just send my autonomous car to go take a picture of something? I might not have done it before, but now I will because it only costs a few cents/mile and none of my time.

Another really cool use case will be sending your car to drive around the block for an hour or go back to your house and park, while you are going out to places. Beats paying the $10 to park for an hour.

Lots of interesting effects there, all the way up to basic urban land-use decisions.

Suddenly the price people will pay for parking is going to be closely related to the price of gas. The maximum value of a parking space in the city is basically the cost to send the car back out to some suburban parking garage during the day and have it come back in later for you. My guess is that the market-clearing price is going to be a lot lower than it is today. Lots of urban parking structures might end up getting repurposed for other uses, if you can't fill them at hundreds of dollars per space per month.

In the short run I could see the average self-driving car's gas usage being higher than a conventional car's, but in the longer run the same self-driving features might make fully electric cars more palatable. Long charging times don't really matter if you can let the car go and charge itself while you're at work, for e.g.

And if that place that it goes to charge itself happens to have a pile of solar and wind, even better! Way easier to do a bit installation in a less populated area than right in the centre of downtown.

Reply on news.ycombinator.com