Tesla’s Push to Build a Self-Driving Car Sparks Dissent Among Its Engineers

Everybody who's looked at this seriously agrees. The aviation industry has looked hard at that issue for a long time. Watch "Children of the Magenta".[1] This is a chief pilot of American Airlines talking to pilots about automation dependency in 1997. Watch this if you have anything to do with a safety-related system.

[1] https://www.youtube.com/watch?v=pN41LvuSz10

What are you measuring? The current autopilot already appears to be materially safer, in certain circumstances, than human drivers [1]. It seems probable Level 2 systems will be better still.

A refrain I hear, and used to believe, is that machine accidents will cause public uproar in a way human-mediated accidents don't. Yet Tesla's autopilot accidents have produced no such reaction. Perhaps assumptions around public perceptions of technology need revisiting.

> Neither the driver nor the car manufacturer will have clear responsibility when there is an accident

This is not how courts work. The specific circumstances will be considered. Given the novelty of the situation, courts and prosecutors will likely pay extra attention to every detail.

[1] https://www.bloomberg.com/news/articles/2017-01-19/tesla-s-a...

That's got to open up some liability questions. You can bet when people die it will be tested in court. You could make a case that Tesla's going to be liable for many level 2 accidents in the long run anyway, so might as well go all in ASAP.

I believe Auto transmission (AT) car drivers are more easily distracted than those driving stick shift or manual transmission (MT)).

But AT makes life easy for so many people, that no one even considers if AT is making drivers less attentive.

A classic example of this, the worst case scenario I have seen in person, is that of a woman talking on a mobile phone in one hand while switching lanes and taking a left turn at a traffic signal. Things like this just wouldn't be possible if that woman was driving a stick shift.

In that driver's case, probably a Tesla like Level 3-5 driving system would be ideal, making it much safer for drivers around.

So, should we go back to MT for everyone or nothing? Just to be under the impression that it's much safer? Or should every vehicle be a level 3 to level 5 autonomous vehicle?

I think people choosing any of the above options will have valid points and research to prove their points. Only time will tell when and how any of those research results continue to hold true.

i heard that this is why autobahns are made with unnecessary curves/turns.

This approach worked well (for some parties) in many other areas, for example in education: now the teachers are the only ones left responsible for the structuraly failing education system.

Now, if these were 1930s, with hundreds of independent auto makers, perhaps the invisible hand of the market would fix this.

https://www.bloomberg.com/news/articles/2017-01-19/tesla-s-a...

I don't see how you can get to L5 without L2 and the learning that is going on in these cars today.

So there is always something to do.

This isn't building a bridge. If a company builds a bridge and it collapses and kills people and it turns out they didn't hire qualified structural engineers or that the CEO ignored warning from the engineers to push the project for a scheduled release window or to keep profits high - the CEO goes to prison for criminal negligence. With self-driving cars, it's a different story COMPLETELY. You're talking about SOFTWARE. No company that's killed people with software has ever found themselves being found guilty of criminal negligence. And they won't for the forseeable future.

This is how self-driving cars will go. I'll give you whatever you want if I'm wrong. I'm that confident. A company, using standard modern business practices (that means doing absolutely every single thing research has shown destroys productivity and ensures the end product will be as unreliable, and poorly engineered as possible. Open floor plan offices, physical colocation of workers, deadlines enforced based on business goals rather than engineering goals, business school graduates being able to override technical folks, objective evidence made subservient to whoever in the room is more extroverted, aggressive, or loud, etc. You know, you probably work in exactly the kind of company I'm talking about because it's almost every company in existence. Following practices optimized over a century to perform optimally - at assembly-line manufacturing processes. And absolutely antithetical to every single aspect of mental work.) will rush to be first to market. Maybe they'll sell a lot, maybe not. That's hard to call. What's NOT hard to call is the inevitable result. Someone dies. Doesn't matter if its unavoidable or not. No technology is perfect. That doesn't matter either. Won't matter what "disclaimers" the company tries to pull trying to say it's in the drivers hands. The courts won't care about that either.

But... they will absolutely get away with it. They will not be fined, they will not be forced to change their practices (most likely they will not even be made to REVEAL their development practices at all). You see, if the courts bother to ask what their practices are, their lawyers will point out it doesn't matter. There's no such thing as "industry standard practices" that you could even CLAIM they failed to follow. So their software had bugs. As far as the court is concerned, that's a fact of life, it's unavoidable, and no company can be held responsible for software bugs. Not even if they kill people.

So they'll get away with it - in the courts. In the court of public opinion? Nope. You see, even if they made their self-driving cars out of angel bones and captured alien predictive technology and it never so much as disturbed someones hairdo, they are destined to fail as far as the public is concerned. Because human beings are, shocker, human beings. They have human brains. Human brains have a flaw that we've known about for ages. Well, by "we", I mean psychologists and anyone whose ever cared enough to learn Psych 101 basics about the brain. There is an extremely strong connection between how in-control a person feels they are and how safe they feel they are. Also, feeling safe is stupendously important to humans. This is why people are afraid of flying. If things go wrong, there's nothing they can do. (The same is true when they're driving a car, but people are also often just wrong and they wrongly think they have some control over whether they have a car accident or not. No evidence suggests they have the ability to avoid most accidents.) If the self-driving car goes wrong while they're not paying attention, nothing they can do. People will be afraid of them as they are of flying.

And if you haven't noticed, our society deals poorly with fear. They LOVE it way too much. They obsess over it. They spend almost every waking hour talking about it and patting themselves on the back about what they're doing or going to do to fix their fears and the fears that threaten others. Mostly imagined fears, of course, because we're absurdly safe nowadays. So it will be the only thing talked about until unattended driving laws get a tiny extension to cover the manufacturing of any device which claims to make unattended driving safe. It'll pass with maybe 1 or 2 nay votes by reps paid for by Uber, but that's it.

Mostly, the drivers, insurance, companies and other people on the road have fewer accidents.

I suspect that if/when LIDAR is cheap enough, Tesla will use it.

In the meantime they outfit every single car with the best hardware that is realistic from a cost standpoint today, instead of waiting til 2020.

I worked on a team that evaluated the ASC unit a few years ago, but they found it unusable due to bloom issues. Has that changed?

I think I agree with this, but is LIDAR expected to work in the rain?

https://m.youtube.com/watch?v=BE2lQK_0CDw

And Tesla's engineers aren't the first to bellyache about being asked to make the impossible a reality.

> Nobody will ride a driverless vehicle provided the explanation that it is, you know, "already an improvement when compared to a typical driver".

I don't necessarily think this is true (perhaps age-correlated?). Let's set aside the issue of whether or not we can do it, for now, and assume that we have a scenario where self-driving cars are safer than human drivers.

In this context, I can easily imagine a political campaign à la "Think of the children!" that paints human drivers as fundamentally unsafe, advocating for a self-driving mandate in urban areas. Perhaps with a Cash-for-Clunkers type of deal to aid the transition. I am not saying this is desirable, merely plausible; it has all the elements of good politics: an easily-grasped bright-line dichotomy, emotional manipulation, and massive corporate benefits (for vehicle manufacturers, self-driving software vendors, and transportation providers like Uber).

I'm ranting here, but you're on an undeservedly high horse.

> If you substitute, you have to do much much much better. Nobody will ride a driverless vehicle provided the explanation that it is, you know, "already an improvement when compared to a typical driver". Is it fair? Maybe not, but that's the whole point for entrusting lives to a machine.

I've heard Elon mention this and while I don't know exactly it's measured, he claimed that fully autonomous cars would have to be 10x "better" at driving than a human before they would be allowed. I'm paraphrasing when I say "better" but I'm sure I could find the video of him talking about it.

At any rate, your comment is interesting.

If human flight was possible, surely they would have done it hundreds or thousands of years ago.

Your argument disregards not only advancement in theoretical knowledge, but also advancement engineering in the form of computational power and sensor sensitivity.

On the bad side, this risk culture destroys large scale innovation, and even safety in the long run. The problem is that we've adopted an attitude that safety is always first, meaning that it is immoral to do something in a less safe way, no matter what the other benefits might be. This means we get a regulatory, tort, and engineering culture that is willing to use existing systems, because they are grandfathered and therefore "reasonable", but will only adopt new systems if they can be shown to be perfectly safe.

This culture is fairly new. I date it to sometime in the seventies. Ralph Nader and the Pinto were both symptoms and causes. You can see the transition in, for example, how America responded to the Apollo 1 fire vs. the Challenger accident.

Since you're a a rail engineer, let's look at rail mass transit systems. The NYC subway has a limit of roughly 30 trains per hour, a two minute headways. All of the braking rates, margins of safety, and signaling systems do their job, and you never see two trains hitting each other. During rush hour though, trains are packed way over capacity, and this is mostly because of this headway limitation. If you were to imagine this on the freeway, you'd have to leave two miles between you and the next car. The cost for this level of safety is that about half a million people have to spend an hour every day in miserable conditions. Many of them choose to drive or take cabs instead, to avoid this.

When they make this decision, none of them even give the slightest thought that driving is maybe 10x more dangerous than taking the train. They understand that safety for both is plenty good, and the way that they spend two hours of their day, 13% of their waking hours, is a lot more important than some tiny difference in their risk of dying on the way to work.

Ultimately, while very well intentioned, this safety culture is inhuman. It's pessimistic. It says "nothing in your life could possibly be so important that it's worth any possibility of you being injured. "

So, this is what we face now, over and over. With self driving cars, it's, "sure, the chance of you getting killed is half as much as if you had driven, plus you get all of that time back to have a nice conversation, read a book, or idly stare out the window, but since it doesn't meet our aviation/rail transit level of safety, you can't have it." You even see it with kids -- your twelve year old kids can't be allowed out of the sight of an adult, or you're a dangerous, neglectful parent. It does not matter that childhood is the process of learning how to be an adult, and that becoming an adult is a process of progressively mastering greater and greater freedoms -- the most important thing is that we are never seen to expose children to even the most minute level of risk.

I really want us to have a real conversation about acceptable risk, informed consent, and human progress. We need it badly to regain our souls.

I am not disputing your assessment, but please don't discount liability. Planes can pretty much fly themselves today - there are no significant technology issues with the idea of "taxi away, take off, fly to destination, land, taxi to gate". all of this happens in what is perhaps the most regulated traffic environment on the planet.

The issue is with creating the code that deals with "oh shit" scenarios. Whilst is is probably possible, and even feasible, to write code to cover every possible failure scenario, who is going to be left holding the can when this fails (all systems have a non-zero probability of failure)?

Who will be held responsible? The outsourced company that coded the avionics/flight control software? The airplane manufacturer? The airline company? The poor fucker that wrote the failing logic tree that was supposed to deal with that specific failure scenario, but was forced to work overtime the 47th day in a row when that particular code was cut?

It is a liability nightmare, and when you add up the cost of creating a software system that must never fail, the increased insurance premiums, the PR/marketing work to convince the unwashed masses that this is actually safer, and the whole rest of the circus required to make this a reality, you will find that pilot costs are not all that bad. Especially since pilots have significant downward pressure on real earnings these days anyway.

It will be particularly interesting if accident blame is placed on the 'dumb cars', and then insurance companies do a 180 and charge MORE for 'dumb cars' operated by humans. Once they put this information in their pricing models, I assume its 'stuck' in there until the next major NTSB report is published.

As complacency sets in over a couple months & years, the accident rates will likely swing from "dumb" human operated machines back to "Level 4 Highly Intelligent Teslas/UBERS/Argo AI", and that market might get a real shock when the pendulum comes back their way!

An advanced AI may well be able to identify whether an object coming towards the windscreen is a bird, bat, leaf or a rock, but what will its intuition be about how much of a problem it is likely to be? Should it swerve to avoid a raccoon and risk whiplash for passengers? Should it aim to avoid large insects if the owner is vegan?

Also, people are very used to mild lawbreaking. We expect a cab driver to double park and let us out the car if there are no available parking spaces, but would an AI be authorised to bend the law or would it have to find the nearest parking space, which may be 5 blocks away and could be taken by the time it gets there?

I suspect we will get autonomous drone-like flying cars long before we get full autonomy in city centres or rural areas, because flying through a mostly obstacle-free space with an ability to avoid collisions on 3 axis seems much more reachable?

Thankfully this isn't as big of an issue with driving on the ground. Airplanes don't have sensors that give them the same precision as a car's wheel rotation or proximity to nearby objects.

E.g. I would quit too.

I am one of the biggest Tesla fans out there. I fucking love the company. But Autopilot in its current form is nothing short of dangerous.

I took a test drive in a Model S a couple months ago and enabled Autopilot at the Tesla rep's encouragement while on a straight stretch of route 90 near Boston. We were going 70mph, a safe speed.

The car came to a point where the highway curved, and a slight deceleration is required to navigate the curve correctly.

Little did I know, Autopilot stays at the speed you set and does not alter it as the environment requires, short of not hitting the car in front of you. So of course it tried to take the curve at 70mph and swung out of the lane almost instantly, prompting immediate corrective action from me to avoid a serious accident.

I couldn't believe the Tesla rep hadn't made this clear. I was required to have my hands on the wheel, but the position of my hands doesn't ensure that I'm mentally ready for egregious errors on the car's part and prepared to correct them at a split second's notice at all times.

Operational question mark aside, as an investor I was also astonished that the software was still in such a rudimentary state that it didn't know to slow down on curves. I found this troubling. It was scarcely more advanced than cruise control, to be honest.

It's the one place where I think Elon is really gambling with people's lives as well as his company's credibility, the former being an infinitely worse transgression than the latter.

We used to treat our test pilots with the highest regard. How low have we fallen?

https://www.youtube.com/watch?v=ZBaolsFyD9I

https://www.youtube.com/watch?v=IOnuKrzCLYc

The previous system (HW1 or AP1) is a MobileEye chip, which is found in most other cars that also have "lane keeping" features. They all use the same chip with the same model provided by MobileEye, and can choose to overlay features on top of it or augment the system with other sensors (such as radar on the Model S/X). For that system, it is essentially the same performance whether it's a $150k Tesla or $30k Honda.

Just driving around New York City for a while makes me think that generalized autonomous driving, as a problem, is essentially "solving" strong AI. Consider the case of approaching a complex intersection during rush hour. There's a traffic cop in the intersection waving his hands around. You reach the intersection, the light is green, you want to proceed straight, but cars are blocking the way because they're backed up into the intersection on the cross street. The cop points directly at you, making eye contact, blows a whistle, and shouts at you, pointing and yelling "right right right". You're uncertain whether he means you should try to weave around the blocking cars, but he blows the whistle again and it becomes clear he is telling you you cannot go straight, and you must divert and make a right turn right at the intersection. He gestures again, indicating that he wants you to turn into the nearest lane on the cross street, then points to the car behind you and indicates that it should also turn, but into the center lane. You nod, and he looks away to another car.

This happened. So a self-driving car would presumably have to understand and interpret shouted commands, realize that they are the one being shouted at by someone with the right authority, recognize gestures, somehow be able to engage in the equivalent of recognized eye contact, be able to make an OK gesture, and have some sort of theory-of-mind about the traffic cop as well as the drivers of other cars.

Not easy.

https://hal.pratt.duke.edu/publications

I saw Cummings speak on a panel a couple years back. She talked about automation in aircraft (she's a former Navy fighter pilot) and she made the comment that humans don't handle boredom well.

Probably quite a good part of the $hundred of millions price of a passenger plane is the autopilot (even 1% means $1M). And even at $1M/plane, 99% of a plane's autopilot works because it assumes that the current plane is the only one in a large vicinity of a point in space. This is assured by a centralized third party (control tower) that is not really automated but a very stressful human job (that's why the air traffic controllers are well paid). This is not the case with cars - in this case, most of the work being done is having each of the individual cars detect, with complex but not very good sensors and software, what is around them, in a swarm of other moving objects that do not communicate.

Lane departure warning and collision warning systems are pretty prevalent. And even autobraking collision avoidance systems are getting pretty widespread.

As we now know, all of them are cheating.

(Also, "having more ethics" sounds really strange in my ears. "... being more ethical than those at <XY>"?).