What Self-Driving Cars See

I'm wondering if we'll ever be able to replace horses with cars.

The thing is I'm often looking at the horses and they are looking at me :)

Self driving on highways is already working. There are solutions for interacting with periastrians. Mercedes had a prototype where a virtual cross walk was displayed in front of the car to show it was waiting for the pedestrian.

Just attach lasers to your helmet and you should be able to divert all the autonmous vehicles away from your path. They're pretty easy to fool right now.

I have to admit that self driving cars are not better in all circumstances. They don't have to reach that bar for me to prefer them though. If they do better in enough situations to outweigh the situations where they do worse, they should displace non-self driving cars. The biggest advantage I see is that they always perform to their standard, whatever it may be, instead of wildly deviating as humans tend to since humans are often drunk, tired or distracted. Their biggest disadvantage is dealing with real life context clues like you mentioned, or times when the road is different than what the map suggests and you need some common sense.

Adding additional methods for self-driving vehicles to signal state to other road users would be helpful in this regard.

"Monderman found that the traffic efficiency and safety improved when the street and surrounding public space was redesigned to encourage each person to negotiate their movement directly with others."

https://en.wikipedia.org/wiki/Hans_Monderman

I remember reading about some design studies for "car-to-pedestrian communication" a while ago. I think one idea was to project a zebra crossing in front of the car to let pedestrians know it's safe to cross.

I think ideas like that show that there might be ways to solve those problems - but interaction with a self-driving car will likely be very different than interaction with a human-driven one. (And you'd have to teach/convince people to accept this new mode of communication)

In any case I agree it's very early days to even attempt to solve those problems.

(Of course it's possible things will go the easy route and self-driving cars will more be regarded like trains and restricted to special routes strongly off-limits to pedestrians.)

What is difficult for lidar are the following:

* not-reflective / dark surfaces, i.e. http://www.thedrive.com/tech/9293/luminar-says-its-lidar-tec... has 10 percent reflectivity at 50 meters. As stated in the article Luminar is able to do the same at 200 meters, but violating eye safety standards. We might introduce paint restrictions - http://blog.caranddriver.com/why-better-paint-coatings-are-c... - but it's not really acceptable to paint everything I think. :-)

* fog and dust particles (already known since lidar use for aerial imaging).

* interference with other cars not so much (much worse with radar).

I don't know if soft/hard objects are so easy to recognize for lidar though! That's pretty challenging!

If you think humans are "two stereoscopic cameras and a limited sonar", you need to learn more about humans.

Not only are our eyes far more versatile than the most sophisticated camera equipment, we constantly maintain a mental model of the world that allows us to focus only on the things that matter, and accurately infer state. This affects everything from vision, to hearing, to proprioception and situational awareness.

Machines don't come close, which is why the quality of the sensors matters so much more. This trope gets repeated in nearly every thread about self-driving cars, and reveals mainly that engineers don't know much about biology.

Limited sonar? I'm not sure what sonar has to do with driving. Between luxury cars with soundproof boxing, or constant road/wind house in a convertible I don't think sound plays any type of role in driving

> "stereoscopic cameras on a swivel" !?

The movement, focus and light aperture mechanisms of the human eye technologically outstrip this description so ridiculously that I have to assume you're joking. Try designing and building a system that can change heading and focus as rapidly as the human eye, while handling nearly as high a input dynamic lighting range. Not even considering the physical space efficiency and impossibly low power consumption of the human eye relative to to do its job compared to any manmade sensors.

AI discussions often get caught up purely in processing and learning, but the mechanical and sensory systems of humans are also non-trivial to integrate into machines, and not to be underestimated.

It's really a question of timeframe. You are completely right that we theoretically only need two cameras. But by limiting yourself to 2 cameras (or cameras + other non-lidar sensors) you're just making the problem a lot harder for yourself. That's a lot more computer vision tasks your research team is going to have to solve, a lot more models to develop, a lot more failure cases (less redundancy).

Some of these vision problems have uncertainty over how long it will take to perform at human-level accuracy. Whereas if you go with lidar, I don't think there's any doubt the costs will go down. IMO on the market, a $5k sensor cost is also nothing if your car is actually fully self-driving.

You are right. Lidar is the right answer _right now_, because alternative requires too much computation. SLAM, Visual Odometry, Structure from Motion, Multiple View Stereo, PMVS, CMVS, or even raw NNs will replace Lidar when computation gets cheap and compact enough.

The more I think about it, the more I am convinced that we humans are _un_suited for driving safely, given the equipment we have. Because we have only "two stereoscopic cameras on a swivel," we have to rely on a complicated system of mirrors -- often at least four (rear-view, two side-views, and a small blind spot checker) -- to get a full view of our surroundings, and we have to constantly swivel our eyes and neck to flip between front view, mirrored views, and instrument panels. We are unable to monitor all of those things at once.

Moreover, we have only an extremely crude way of communicating with other drivers to coordinate our actions. We have two turns signals, a brake signal, hazard lights, and a horn, none of which is guaranteed to be used appropriately. A lot of traffic jams and crashes could be avoided if our vehicles could broadcast their intended actions to each other in greater detail and with more reliability.

Crash data certainly bears out the fact that humans are not well suited for driving. While you say "We already know that you can be successful at driving a car in various weather conditions," I would instead say that in the aggregate, we do better than one would think, given our constraints, at avoiding crashes -- but we humans crash a lot. Millions of times a year in the U.S., killing tens of thousands, and resulting in some sort of injury in about half of cases.

So, whatever technology is being considered for self-driving cars, comparing it with the "technology" of human drivers is a natural baseline, but I would think that the greatest gains would come from breaking away from the equipment limitations humans have.

No. The bar for new technology to meet is to improve upon safety outcomes, not maintain.

That's kind of Elon Musk's take as well:

>“Once you solve cameras for vision, autonomy is solved; if you don’t solve vision, it’s not solved … You can absolutely be superhuman with just cameras.”

though I can see the argument for using a bunch of other sensors as well especially given that current self driving AI is a lot less smart than humans.

I'd say that using "base stations" or "turrets" like in airports wouldn't make a autonomous-self-driving technology.

I guess I prefer a fully independent car capable of driving anywhere without the help 3rd party devices.

Ok maybe just gps.

That sounds like a centralized approach to the problem. Which ive thought about, too. If the government provided more infrastructure to assist driverless cars it would probably accelerate adoption and the installation and maintenance costs of LIDAR turrets and other infrastructure would ideally be offset by the number of lives saved. Im also wondering if a something similar to air traffic control could be applied to self driving cars.

You could avoid interference using techniques similar to ones used in direct sequence spread spectrum - rather than a continuous beam, send out pulses in a pseudorandom sequence, and look for that sequence in the reflected returns. With each transmitter keyed differently you can ignore reflections from other transmitters.

Pretty sure this is anthropomorphizing too much. Why look for the reflection when you can literally detect around the corner? More, if the system is fully automatic, just regulate the speed so that it can stop in time for events like these.

okay, you drive those 10 seconds until you're on the road and it'll do the rest.

I don't know the exact state of recognition used in the state-of-the-art self-driving cars, but I wouldn't you be able to handle reflections if you use normal camera image recognition combined with LADAR? From a simple image, you can infer the distance to an observed object by looking at the size of its bounding box, and compare that to what the size of a bounding box of that type of object is at some known distance. After that, you can validate this distance using LADAR. If there are large discrepancies, you can either infer that it is a reflection, or you can maybe have trained a system specifically to classify if something is a reflection or not. That system would probably look at how much distortion there is on an object, compared to the rest of the scene, which could be caused by e.g. a non-flat reflective surface.

Like I said this is just some thinking out loud on my part, and I am by no means an expert. Does anyone that are more knowledgeable about this topic know if this strategy could be feasable?

Can Lidar distinquish between an empty plastic white bag dancing on the wind in the middle of the highway, and a baby that fell out of a stroller?

In both scenarios whether it can or cannot, its a distaster.

From the comments: "It has been brought to my attention, that some of these clips may be fake. A new video has been made that eliminates the fake clips https://www.youtube.com/watch?v=rphN3R6KKyU ".

Not to detract from the fact that autonomous driving has the potential to be safer than humans (if only because it doesn't get distracted), but none of these autonomous responses are "freakishly early". I've seen attentive human drivers respond with similar or better performance. As a matter of fact, let's examine how human drivers responded to one of the collisions in this video.

At 00:28.828, a human could tell that an accident is imminent -- the oncoming car will violate the median should it not apply immediate corrective action. However, other drivers are not obligated to wait until the aggressor vehicle crosses the median to take evasive action! Indeed, the silver vehicle (ahead of the dashcam'd vehicle) begins decisive and correct evasive action at 00:28.328. That vehicle maintains the same steering angle and at 00:29.295 crosses the solid white shoulder/"fog" line -- it's unambiguously an evasive action and absolutely not a normal lane change.

The autonomous vehicle only begins to brake/turn at 00:30.130 (the first frame where the car begins to pitch forwards due to the weight transfer). The autonomous vehicle has begun to take evasive action a full 1.8 seconds later than the silver non-autonomous vehicle. Awkwardly, the Tesla's reaction seems to be triggered not by the median-violating vehicle but by the white vehicle (who is attempting to take evasive action) cutting it off. Worse, the silver vehicle is already crossing the shoulder line a full second before the Tesla begins to brake/turn. This isn't even commensurate with human performance -- the two human drivers visible in the dashcam video react significantly earlier than the autonomous vehicle.

It's absolutely possible to make vehicles that respond quicker and more effectively to impending collisions than humans do. Sensing with radar and IR can give autonomous vehicles information that human eyes simply can't detect -- IR is useful for detecting human presence, for example, and radar can be used to detect positions of cars beyond-line-of-sight. Better sensing translates to earlier warning. Also, when the computer has full control authority over individual wheel torques there's a whole new range of operating regimes that get opened up -- and autonomous vehicles that exploit those have the potential to make cars much safer (see research like http://news.stanford.edu/2015/10/20/marty-autonomous-delorea...). This sort of tactic can use the tyre grip to the absolute limit can let the computers craft the optimal (given constraints such as the vehicle's position, surrounding vehicles/barriers, and tyre/road conditions) trajectory out of a collision and execute it with precise per-wheel torque control (via the ABS/ESC system) and high-speed steering input. Even the best drift drivers don't have that level of control authority over their cars!

There's lots of room to make autonomous vehicles safer than human drivers but this video is a demonstration that autonomous vehicles can be a lot worse than human drivers.

To my limited knowledge, they don't. Deep nets are mostly used by the self-driving companies for processing camera data for image detection and segmentation tasks like detecting other vehicles, recognizing street signs, etc. Presumably, having 3D point clouds lets you hand-code solutions to a lot of these tasks without having to use ML.

For all the non-regulation of US roadspace, I don't see any significantly different driving in Europe: having driven enough on both continents, it feels the same w/r/t unpredictable drivers. IMHO it doesn't matter whether the unpredictability comes from "has no training" or "has training but ignores it".

LIDAR is supposed to have low attenuation in water, however it will still suffer refraction. Hopefully it will function about as well as a human. The vision processor will have to understand that it's raining and correct for refraction where possible.

Snow is a different challenge, but a sufficiently smart system should be able to composite frames and appropriately fill in the holes created by being obscured by snow. It won't be trivial, but it should all be doable.

Yes, kinda https://qz.com/637509/driverless-cars-have-a-new-way-to-navi...

Perhaps because machines can't be held responsible for deaths they've caused. Though labelled 'autonomous' theirs is a somewhat lesser form of autonomy than you or I possess.

It's an argument, IMHO not a very good one.

It's basically the same as "but they don't cure cancer!" argument.

Coded pulses: rather than emitting a uniform beam, they emit a pulse and listen for that back. You can think of it a little like a bar code, a sequence of short and long, on and off. Then select for that on receive.

Nobody likes to talk about it, but they don't. It turns out there are some clever tricks you can do if you control all the cars, but if not you have to get competitors to agree to some protocols.

Much the same way two routers broadcasting different WiFi networks in the same space and received by two different computers don't really interfere with each other. It's all just EM stuff, you can apply standard EM communication protocols like modulation. However standards will still be necessary, it can indeed be messed with.

Lidars are highly directional and have low duty cycles so even when large numbers are present the amount of interference is small. So you do have to model a certain amount of noise in your lidar returns but you have to do that anyways due to things like sun reflecting off of shiny surfaces. Noisy sensors have been a fact in autonomous robots for as long as there have been autonomous robots and there are a variety of standard techniques for coping.