Andrej Karpathy (Tesla): CVPR 2021 Workshop on Autonomous Vehicles [video]
youtube.com
youtube.com
There is absolutely 0 chance they could just 'put in lidar' from now on so for what he has to do its not relevant.
Multiple companies that have announced Lidar have later removed it for the cars.
The Lidar that are actually currently deployed that you can 'just' get from suppliers is of the not of the same quality and we have yet to actually see real high volume cars with 360 Lidar on any true mass market car.
People keep getting hung up on their vision strategy because it sharply diverges from that of all other players, but it's really just tackling the "no one knows how to do this and no one really tries" part of the problem first.
No level 5 autonomous driving system will ever ship without really good computer vision.
Is LIDAR required for self driving? Probably, but it's not certain. But one thing is certain - Tesla is in the corner, and they have to deliver solution without it, using few years old inference HW and 5 years old+ cameras. Selling HW for a problem that doesn't even have a theoretical solution in software, so you don't even know what HW you need, is a bold, bold move.
Also, the sensor resolution issue he mentioned could be addressed by using a higher resolution radar sensor.
I find the list of 221 triggers to be interesting. In principle, the NHTSA or NTSB could help contribute lists of triggers to companies to validate their training sets on.
Every time there is a fatal airliner accident, the NTSB does a safety investigation and airliners get a little bit safer each time. In the same way, each fatal accident in a vehicle with this kind of autonomy could end up being captured by these triggers, improving safety over time in a sort of mixture between expert human analysis and ML.
(Nobody does this for all regular car crashes because fatal car crashes happen every day! And you’re not going to retrain human drivers about some new edge case every day, although you can for vehicles like this.)
You sparked a happy delusion in my mind...training drivers to the same level we train pilots. Can you imagine drivers having regular check rides?
They seem to use radar solely to automatically label data for training.
In the given example though where according to the vision system smog interrupted the persistence of the label for the leading car, I wonder if the use of radar data to persist the label is strictly necessary.
A car disappeared then reappeared in the data, why not just tween the bounding box over time and assume the car had always been there, like, if it looked the same when it reappears or something. An extra sensor just for labelling data seems silly.
The argument may be that humans do fine without mapping, but that's not strictly true. Humans remember and anticipate the road based on past memory. When there's a change in the road due to construction, there's often a sign placed to warn drivers of the change in traffic pattern, showing that humans definitely at least partially rely on "mapping," so it could help even a pure vision system.
Prediction: Tesla will be the last of all major auto manufacturers to get to L5 autonomy. Time interval between when Tesla L5 FSD is finally available and when humanity is destroyed by the general AI it runs on will be very awesome and also very short
Even if you believe this goal is possible to achieve at some point in the future, I think the argument falls apart when you consider that it will take years, probably decades, for a pure vision approach to catch up to where Waymo is today in terms of safety. (They have cameras too.)
That Tesla can’t afford to fit expensive LiDAR sensors to all of the cars it sells is Tesla’s problem. Regulators won’t give a shit that pure vision is “better” in theory. They will simply compare Tesla’s crash rate in autonomous mode with that of Waymo and other AV operators, and act accordingly.
However, nowadays it starts to look that 100% reliable depth estimation from cameras might actually require a human-level AI to work and also solid-state LIDAR technology is becoming cheap enough and integrateable into normal cars, but Tesla can't really change their stance on this without admitting that FSD options they already sold would not actually become FSD within the lifetimes of these vehicles. I suspect this might also be the reason why Karpathy looks more and more nervous with each new talk
That's pretty much a given at this point but they will not admit it until a class-action lawsuit forces them to.
you don't need 100%, and even humans are far from 100% (500Mpx resolution of our eyes allows to basically sheer brute force through it in many cases). The stereo setup provides great and fast estimation with several megapixel resolution with good fps (way better than lidar) for majority of situations. It is some share of the [part of the] scenes, and you really know it right then and there, where you need AI and/or very sophisticated compute heavy algorithms. So instead of throwing AI and the compute power at those parts, you just pull the points from the lidar (and even radar if the things are that bad) covering that segment. And that way, given a couple more iterations of sensors (from current 20Mpx+ to the hundreds Mpx) and compute, it will be doing even better than humans. Anybody not doing sensor fusion would be a loser though - just like going into a fist fight with one hand intentionally disabled.
On what set of metrics do you think Waymo is safer? IMO it's too early to compare and cherry-picked proofs both from Waymo and Tesla are not really representative.
You can read their other safety whitepapers in https://waymo.com/safety
That doesn't work when they scale up to any other set of normal roads, especially as density and complexity increases.
They will always map roads to precision, whether it's Arizona or San Francisco. Why is that a problem? You should either look at their CA disengagement reports over the years or wait until they roll out a service in SF (where they've been testing heavily). That will show how safe they are in dense environments.
The video with the Waymo car getting stuck and taking off from the rescue team had an example of this.
I guess it makes perfect sense from a engineering perspective.
There are issues with reading too much into disengagements, but there certainly seems to be a large difference here.
[1] https://thelastdriverlicenseholder.com/2021/02/09/2020-disen...
Tesla wimps out and won't test in California, because they'd have to report.
For example, cameras are sensitive to glare from reflections (sun near sunset or reflecting on metallic objects) and oncoming traffic at night. Lidars operate on a different narrower wavelength and are unlikely to be affected by that, although they might struggle with objects that have low reflectivity at a long distance.
The fact that these sensors are different means that the intersection where a dangerous situation would not be detected by either sensor is much smaller than any sensor individually.
In any case, once AVs are deployed at scale, if it becomes apparent that some sensors can be removed or replaced by something else, then they will be if there's a case for it.
Tesla is also the only company to claim to target L5 autonomy. Everyone else, including Waymo, is strictly targeting L4 and say L5 autonomy is not possible or realistic. L5 is a pipe dream.
Vision can also get you 3D maps, either in active manner (IR floodlight or structured lighting) or not. I will reserve my judgement until see more from either side.
I am actually on sensor-fusion side, but I don't think we should jump to the conclusion LIDAR is the best 3D mapping method.
For one example, a truck has a breaking distance of 600ft, either Velodyne or OS1 LIDAR has range less than that.
Take a look at how dense the point cloud from Waymo's 5th gen LIDAR is: https://www.youtube.com/watch?v=COgEQuqTAug&t=11599s. They just talked about this a few days ago.
In effect, they solved the disagreement problem by sticking their head in the sand and pretending their sensors are never wrong.
I am actually on sensor-fusion side, and think a transformer can merge everything and generate a coherent world-view. But this is a hard problem and people side-step them after the evaluation shouldn't be dismissed blatantly.
For one:
> It gives you a more robust view of the world than using a single sensor modality.
How can we evaluate that correctly?
If anything, knowing when to reliably ignore a sensor modality is the kind of intuition more associated with general AI.
A similar paradox occurs when trying to fuse multispectral imagery. You'd think early fusion of RGB and IR would be better since it gives the higher-resolution filters access to more data, but it does worse than late fusion. My understanding is that late fusion forces the network to "work harder" to solve object detection using IR only, and then once you've wrung what you can out, then you fuse with RGB detections.
Since radar is "one pixel" there's essentially only one object detector possible: object or nothing. If yes-object, fusion tries really hard to make sense of the RGB filters to figure out what partial detection looks like an object, which is almost always a false positive.
Waymo has its own custom developed 4D imaging radar and I imagine most of the other SDC players have their own versions.
[1] https://blog.waymo.com/2020/03/introducing-5th-generation-wa...
[2] https://aurora.tech/technology/driver
[3] https://www.mobileye.com/blog/radar-lidar-next-generation-ac...
Given this post is about radar usage in self driving and how Tesla/Karpathy touts their capabilities, I compare them to other SDC players, not other car manufacturers who are not in the game. And their SDC competitors are running away because they have the newer 4D radars.
Only if you don’t consider Waymo, Cruise, Mobileye and others as Tesla competitors.
+1 to the other commenter (grandparent comment).
Radar has a range resolution and an angular resolution. Andrej completely omitted this fundamental aspect in his talk.
In my humble opinion, Andrej needs a real Radar engineer in his group before making such a statement in his talk.
Nobody can be smart for all topics, which makes all humble also.
That sounds like a problem with the network's architecture and not the data itself
This makes some important assumptions, namely that Tesla built a lidar and radar perception pipelines and sensor fusion of equivalent quality to their competitors, and then decided they were unnecessary.
Given that their competitors have shown substantially better perception than Tesla, and that Tesla has a significant economic incentive to deliver autonomous driving on a sensor suite that already shipped years ago, I find that difficult to believe. Did Tesla build good enough perception to dismiss lidar and radar purely on their merits. Unlikely I think. Did an intern build a student-quality lidar pipeline that "proved" Elon's camera-first approach is the right one? More likely.
This is actually my main criticism of Tesla's approach. They don't seem to have enough cameras to do the job well, and its showing in a lot of actual system limitations.
Now it’s an indefensible position on safety or economic grounds.
Obviously, pure vision is a viable system, as it's what we as humans use. The question remains as to whether or not it will be comparable to more precise LiDAR based systems in the near future.
The lidar/no lidar discussion is a fun one because people have different ideas about how the world works. Personally I think LiDAR is the modern version of expert systems. It appeals to a logical/geometric intuition but the approach is brittle to real world contact, especially when paired with HD maps which are a great way to drive yourself into a local maximum.
Having training data beyond a certain point is overrated, and Tesla's advantage in gathering it is overstated. Other companies are capturing this data as well. Is there any indication that the data Tesla is collecting is of a higher value, or is it just more bytes?
It may be valuable enough to require a few petaflops less computing power.
The notion that Tesla's model HAS to keep improving simply because they will be able to pile on more (unlabeled!) data is laughably false. And, in fact, quite insulting to the intelligence of even the most casual ML engineers.
The point of the higher scale isn't just more data, it also makes it easier to solve the unbalanced data problem, because rarer and rarer scenarios will appear in large enough numbers to work with.
Exactly, casual ML engineers. The issue of plateauing tends to occur because there is no more novelty to be had in the data. What mega-experiments like GPT and similar have shown us is that actually you can keep adding novel data and keep improving the model. Kinda inelegant, yet effective. The problem is, most institutions can't add more novelty beyond a certain scale, since that usually means shoveling more money at data storage and compute, on top of the novelty collection.
Tesla merely has to open the money tap to get more of both compute and storage, and let the real-time data flow in.
And if you watch the other parts of the presentation, you'll see the bits about them buying clusters with 5k+ A100 GPUs. Presumably they intend to do something with those. Probably not streaming Fortnite concerts.
given the nature of that data you can get a lot of unsupervised mileage, so to speak, out of it.
A team with funds like Tesla, Google, FAIR is going to be using NAS and have a continuous testing pipeline. Tesla has arguably the best environment for continuous testing which is the most difficult part of improving a model. Andrej even said in his talk that their supercomputer is in the top 5 for FLOPs.
SOTA on ImageNet for the past few years has been driven by pre-training on massive datasets. Vision transformers are increasingly more common and are extremely data-hungry.
That the exact opposite of what they are doing.
It seems like you didn't watch the talk at all.
You can only go so far by dumping more data into it. Diminishing returns.
If Tesla was developing their system with trained safety drivers or on closed courses, I think they would have higher moral ground to gamble here. But placing the untrained public behind the wheel of alpha quality software is unethical IMHO. There ways to develop autonomous software that are significantly less risky, and the only reason Tesla is doing it this way is for marketing/PR purposes as far as I can tell.
Right now their profit model is selling cars, not autonomy, so everything is optimized for that, including the decision to not use LIDAR.
At some point when LIDAR is cheap enough there is no reason for Elon Musk to not give in and use them. Right now he's constraining the problem to the cost of the car.
I think that's largely an issue with the early LIDAR devices today, but not necessarily what may be to come.
There's something to be said about measuring actual data with solid physics vs. inferring distances with billions of operations on RGB data. If you were landing a commercial aircraft in fog, you most certainly don't rely on your eyes to do most of it, but it is in fact possible to do safely precisely because we do have good sensors on them.
I fully agree with leveraging the scale and maturity of RGB sensors for cars today, the talk is spot on about that, but that's (a) circling back to the fact that Tesla needs to sell cars now not next year and (b) not a good case against use of LIDAR in the future.
As mentioned in this talk Tesla is taking an iterative approach and trying to make things safer for people today, not in 5 yrs from now. Maybe in a few years Tesla will see that to go from 10 million miles to 20 million miles they need 4k cameras at 60 fps, but the work they had already done would still have had a big impact. You don’t need to do things in one shot and get to the finish line.
> You don’t need to do things in one shot and get to the finish line.
I think we agree on these things. I wasn't trying to dispute their approach, rather explain their rationale for their current method, which is focused at actually selling product now.
> But $20k on a $40k car is too much
Those sensors don't need to be 20K-40K, they just aren't mass produced yet. That's why I think Elon Musk will change his tune in some years when those sensors are 1/100 the cost and 10X more accurate than vision.
The criticism of autopilot is really about it getting tripped-up in response to statistically rare, unusual circumstances, i.e., edge cases. Karpathy et al are working on getting better at those, bringing the rate of situations that surprise autopilot closer and closer to 0%, even if it can never be achieved -- there will be always be surprises. Personally, I would rather take a tiny risk of crash on rare, once-in-a-million-miles events with autopilot driving than a ~1% risk of crash per 1000 to 2000 miles with everyday human driving.
Prediction: Tesla will be the first of all major automakers to get to level 4 and 5 autonomy.
Still not close to good enough for people to accept:
"Participants from both countries required Self Driving Vehicles to be 4-5 times as safe as Human Driven Vehicles" [0]
https://www.tesla.com/VehicleSafetyReport
> In the 1st quarter, we registered one accident for every 4.19 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot but with our active safety features, we registered one accident for every 2.05 million miles driven. For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles.
It is not from your data alone. It maybe, but the data is not from a controlled experiment.
Because:
1. Autopilot disengages in dangerous/ambiguous situations. 2. The set of users with autopilot is different from those without.
Lies, damn lies, and statistics.
Tesla here, is, again, being funny with the numbers. They LOVE to cite autopilot ON death statistics as being "10x safer than normal driving". What they fail to note is that Autopilot can ONLY be on while driving on a limited access highway. Highways are much safer to drive on than a mix of ALL ROADS, which is where the baseline figure comes from.
Another confounding factor is the price of the vehicle. The average CONFIGURED Tesla with the FSD package today costs what? $65k? More? Those X's and S's are $100k+. Nobody is buying that base Model 3. The point is that Tesla drivers are 1) Older and 2) Wealthy. Wealthy, older people get in far fewer car crashes than the average driver. In fact, car crash fatalities are really driven by two groups: drunks (or pill addicts), and young (teenage) men. Not saying it's IMPOSSIBLE to have a substance abuse problem and own a Tesla, but the average Tesla owner is less likely to have these issues. It's also less likely to own a Tesla while young.
So, Tesla autopilot stats should be compared to other comparably priced vehicles while driving on the highway ONLY. That would actually be a fair, honest comparison. I believe a recent outgoing BMW 5 series chassis finished its entire life without a single fatality in the US. That's right -- 4-5 years of service in the US without a single death. Turns out, wealthy people who drive expensive family sedans don't get in a lot of fatal highway crashes.
Here's a Forbes article (sorry) doing some of the back-of-the-napkin math. They estimated that in Q3 2019, autopilot really wasn't any safer than manual driving.
https://www.forbes.com/sites/bradtempleton/2020/10/28/new-te...
I disagree. Autopilot driving on the highway should be compared to all human drivers driving on the highway. Otherwise you wouldn't be comparing against human performance per mile driven apples-to-apples.
The BMW F10 535i had ZERO FATALITIES over its entire life in the US. Zero. It had no "self-driving" capabilities. Just lane-departure warning, BLIS, and ACC.
Yes, all evidence I've seen indicates that cars partially driven by computers (adaptive cruise control, lane-departure warning, blind spot information, etc.) are safer than cars entirely driven by human beings. The BMW you mention is safer precisely because it is partially driven by machines. The more we automate driving, as machines get better and better at it, the safer we will all be on the road.
Emphasis on PARTIALLY. Anyone who has read recent takeover scenario studies is rightfully horrified at the notion of a completely “hands off” driving experience, where the driver is expected to remain alert and vigilant but they’re not inputting any steering, throttle, or braking. Unsurprisingly, it takes people about 2 seconds to re-engage as active drivers. 2 seconds is way too long, which is why it may be safer to NOT use a system that does steering input in addition to throttle and brake. You need to keep the drivers actively engaged. And no, “touch the steering wheel every 30 seconds” is not active engagement. And if a car has “self driving” but also active driver monitoring, what’s the point? The driver doesn’t get to relax at all. The stress of driving doesn’t come from the input unless you’re racing. The stress of driving comes from having to stay alert. If I have to stay alert, I’d rather just drive myself instead of trusting an experimental system that drives like an indecisive, half-blind grandmother.
The BMW was safe PRIMARILY because it’s a well-designed, modern car, driven by an older and wealthy (safe) demographic. The assistance systems are probably secondary. They weren’t even standard on all vehicles and they were very primitive in that first generation.
If you actually read the Forbes article above, the back of the napkin math actually DOESN’T indicate that Teslas in autopilot are safer than normal driving. That’s the entire contention. I do not think these full-takeover systems are safer at the present time than active human drivers in comparable vehicles with safety assist systems. Tesla is very clearly fudging the numbers to make it appear as if autopilot is safer, but the claim doesn’t stand up to some really basic analysis.
This isn’t true anymore
I have to give it to Elon that he was able to keep his fans believing a big lie of their FSD system to achieve Level 4 / 5 autonomy last year, when this year it was admittedly Level 2 [2].
The fans will continue to believe his lies and keep on saying "It's coming soon, you'll see..." when they have just been sold a Fools Self Driving System™.
[0] https://twitter.com/Model3Owners/status/1406002366923612163
[1] https://twitter.com/comma_ai/status/1406304017400012800
[2] https://www.news18.com/news/auto/teslas-full-self-driving-cl...
That doesn't mean much considering no company is getting to L5 autonomy likely in our lifetime and possibly beyond.
How can you make such a strong statement when you simply don't know how to achieve full autonomy? This reminds of the teapot orbiting the Sun argument [0]. You and people defending Lidar by their teeth don't sound too different from religious zealots who've "seen the light".
Previous generations of LIDAR were not great at classifying small objects and road debris but it works great for detecting cars, pedestrians, etc.
https://paperswithcode.com/task/3d-object-detection
Next-gen LIDAR has great density, and I bet it would do a decent job differentiating between a plastic bag or a rock in the middle of the road. In addition to depth LIDAR also returns intensity and several other metrics, which can be used as input to an ML model. It's why you can read the lettering on the side of the semi truck in this video of Waymo's next-gen LIDAR.
If the only way it becomes practical to achieve L5 is to use LIDAR, Tesla can obviously add it. But if they waited until LIDAR was cheap and practical, they still wouldn’t be shipping any hardware doing autonomy today, and not collecting the data needed to train their models and delivering value today.
Also, with vision based systems, it operates in somewhat an intuitive fashion given we have eyes too.
It is also pretty easy to see that Tesla doesn't have to hit L5 to have won autonomy. It just has to successfully monetize L3/L4.
That is likely close to true with lidar as well. See also some of Waymo's recent struggles in unexpected construction zones.
Maybe lidar helps in getting there, but I'm afraid they all hit a pretty tough ceiling without this.
That example has nothing to do with lidar, but with planning. The car was able to detect the construction zone just fine.
Examples: human eyes will have trouble adjusting to the sudden darkness of tunnels so some people will tend to brake suddenly; that person looks old and will probably have slower reaction times so watch out for the upcoming sharp turn; that person looks like they are on their phone and may cross the lane suddenly; watch out for this intersection because young humans cross it after school without looking so slow down below the speed limit.
This human understanding doesn’t seem to be directly represented by the system without explicit architecting on their part. A more general intelligence would begin to automatically learn these. A human intelligence would automatically model these or learn from experience or read about it.
As mentioned, the current system has some advantages over humans: more eyes, doesn’t get tired or distracted, faster reaction time. I guess we shall see when these advantages cover up the disadvantages.
Suppose they eventually have this current system dialed in and they get really good, accurate bounding boxes around all interesting objects on the road.
So now, in addition to their 10 second samples of video data they're collecting, they start collecting 10 second samples of scene representations.
These samples of scene representations are time series of how various objects in the scene are moving and behaving over time. Many examples of just what you describe: cars with older drivers having slower reaction times; cars with distracted drivers driving recklessly; etc.
Now you train a model on all that data, asking it to make predictions through time. It's going to quickly pick up on the same or similar cues that humans do. It sees an older person in a car and says that slower, cautious paths through the scene are more likely for that vehicle. They see a large, lifted truck and assume a 90% probability of a "cut off every car possible" path through the scene. Etc.
So I see what Karpathy is building now as a foundation upon which they can build the higher order stuff.
(Surprisingly, he basically ignores night driving.)
I still think that this is far the best demonstration of autonomous driving to date https://youtu.be/A1qNdHPyHu4
- car stalled in it's lane - complicated intersections - people exiting cars in it's lane - car going over into it's lane
If I had an hour of driving that I'd be stressed.
1) The car fails to accelerate to beat the truck as it merges (the safest option given the scenario, even with the yield sign)
2) The car almost collides with the trailer of the truck and it stops with its nose sticking out into the merge lane
3) The car sits there with its nose in the merge lane as other cars go by
4) When the car finally has an opportunity to merge after having done everything wrong up to this point, it sheepishly merges and takes forever to get up to speed. This is basically the least safe thing it can do when yield-merging into a high speed lane.
LOL. This is the best the autonomous driving world has to offer? 80% of this driving is an immediate failure for a teenage driver taking their road course. I think I can take pretty much any random 2 minute sample from that video and find more than a few inputs/maneuvers that would lead to failure in a driver's license test. Why would ANYONE with a brain put their life into the hands of this system?
OH, and do you want to know a little industry secret about these "unedited" long videos? Yes, the video is uncut. But guess what? They drove that route hundreds of times and only showed the best run. And they got laser-HD maps of the route that they won't have globally. This is what Zoox did for their infamous demo that got them bought out by Amazon. And it's what Tesla did for their "Full Self-Driving" video on YouTube that shows driving ability that their cars cannot match even today. Shhh, you didn't hear it from me.
A good driver accelerates on that ramp on beats the truck by 2-3 car lengths and merges at speed. Yield does not mean stop. A cautious driver slows before the merge point, maintains a roll, then mashes the accelerator to merge safely at next opportunity. A terrible driver (Mobileye) stop abruptly at the end of the merge junction, almost hits a truck, leaves its front end sticking out into the oncoming traffic, then dangerously merges sheepishly without accelerating fast enough. You can even see how much this freaks out the human passenger. He thought he was going to get hit!
No. A good driver stops at the yield sign if an approaching vehicle is 2-3 car lengths away. It is dangerous to play guessing games with your car's engine and another driver's attention. If anything goes wrong (tire puncture, engine dies, slippery road, truck driver slams the gas) you have successfully managed to put yourself into the direct path of a speeding brick wall 5x as heavy as you. There is never a safe merging scenario where the right choice is to slam the gas pedal to beat another driver to the punch, especially not when merging onto an active freeway.
If there is a yield sign at the end of the onramp, that means prepare to yield. If there is not a yield sign, then the expectation is to keep merging. There wouldn't be a point to the yield sign if everyone treated it as if it didn't exist.
> leaves its front end sticking out into the oncoming traffic,
I also agree with you but this may be a byproduct of the camera's perspective and maybe it would look normal in the driver's seat. We don't really have a great view but the truck could also have been hugging the curb which would have brought it closer to the car than it should be.
> dangerously merges sheepishly without accelerating fast enough
Fully agree here. If you merge you have to commit to matching the speed of nearby vehicles or else you're creating a dangerous situation. Mobileye should have sped up much faster than it did.
I'm not a neural network, fwiw.
https://i.imgur.com/irkHBxc.png
I think people lower their expectations for autonomous systems. Driving the ramp at a speed a human would and taking that obvious gap to zipper-merge is the correct maneuver for a human. Perhaps you want the autonomous system to err on the side of caution, but I'm actually trying to hold it to a human standard.
But yeah, at the very least, don't get confused and sit there with your nose out in traffic. What's really scary is that this is the take Mobileye chose to go with. They probably had dozens more, with even worse blunders.
Loss of colours in your visual imput as well is a bit of an issue...
Here's a direct link:
This is the right thing to focus on, as it is by far the largest issue with Autopilot on the highway. Multiple people who do testing of these system that false positives on some highway overpasses are the biggest usability issue.
You can't just pull out your street view footage from 3 years ago.
[0] https://www.geoweeknews.com/blogs/google-putting-lidar-new-s...
[1] https://www.androidpolice.com/2021/01/16/google-maps-is-roll...
Musk and Karpathy have been dead wrong about LIDAR for years. Remember Musk making the absurd claim of a million Tesla robotaxis by 2020? I think most hilarious is that both Karpathy and Musk claim the LIDAR systems are too expensive. Yet, in the same 2019 Autonomy Day they simultaneously claimed that Teslas would be able to drive themselves and operate as robotaxis, earning their owners passive income and therefore justifying significantly increased MSRPs. So, the $7k LIDAR system (that accelerates safe autonomous driving) is not worth the cost, yet stumbling towards autonomy on vision only is? If the car becomes an money-earner, you should use all of the systems available. The 2019 Autonomy Day was an utter embarrassment. I'm sure 2021 will be more of the same.
So now it seems that they've realized their folly in logic. So what's the solution? Well, you can't just complain about COST of non-vision perception systems. Because, as noted above, that doesn't make sense if you're going to simultaneously claim that your car will be able to earn you money (augmenting any extra hardware cost that gets you to that point faster). No, now you have to smear all non-vision perception systems. You have to say that their data is worthless and detrimental to the overall effort.
The entire claim from the 2019 Autonomy Day that "vision is what humans use to drive" is also completely bogus. Humans use many senses to drive. They feel the pedals and steering wheel. They use their equilibrio sense to sense motion. And they use their hearing to hear other vehicles, sirens, and issues with their own car (driving with headphones in is illegal for a reason). Any modern car, even a Tesla, is also using far more than just vision when attempting autonomy. Forget about radar and LIDAR for a moment. There are endless sensors in the drivetrain. Steering angle sensors and multiple IMUs for the electronic stability control. Brake and wheel sensors for the ABS. Temperature sensors everywhere. And countless other ECUs. The notion that vision is getting you there exclusively is nonsense. There's no good argument against LIDAR today other than perpetuating a lie to sell cars that are cheaper to produce. And, Karpathy has a massive professional conflict of interest in making CV the main player -- he's a CV expert. He was never a fusion expert before his hire. If CV is the pathway forward, he's gets to remain "the guy". It certainly behooves HIM to make that claim.
Autonomous driving will not be achieved in this decade. Perhaps ever. Ask yourself honestly: if you were tasked with building an autonomous commercial aircraft OR an autonomous car, which would you choose? Most would say aircraft -- nothing to really hit in the air, fully mapped airport and runway systems, and far fewer variables. Yet autonomous aircraft still do not exist. Perhaps the edge cases always rule the roost. Ask yourself why driving would be any different...
Sirens are primarily a means to get you to look in a direction. 360° cameras can notice the emergency vehicle as soon as it's visually relevant. And if they decide they need an audio siren detector, that's like, practically intern level signals detection at this point. Hardly a dealbreaker.
100% hands-down would pick autonomous car vs airplane. Flying a plane isn't just moving the aluminum bird through 3 space and periodically taking off or landing.
Autonomous aircraft don't exist because a huge amount of the ritual of flight is before and after the captain is even on the plane, let alone flying. There is a tremendous amount that the pilot and copilot go through, on the ground, before taxi, after liftoff. It's way, way more involved and way more generally intelligent. We can design AI to take off, path to a destination, and land. Those 3 things are the easiest parts of flying, yet do not comprise the act of flying a 2-seater, let alone an airliner.
I understand the complexities of modern commercial flight. I still consider it far less complex (computationally) than driving on today's public road. The fact that you have V2V communication out of the box (via transponders) is probably the biggest factor. I would not rule out autonomous driving WITH a standardized V2V / V2E system. Without one? I'm bearish.
Having LIDAR means you can rely on LIDAR, which means you need the LIDAR to be 100% reliable. I've read some companies claiming their lidar works through rain and snow but I haven't seen actual renders of it yet to judge for myself.
No lidar acts as a forcing function for Tesla to get their vision system that much more reliable. I think this will ultimately result in a better vision system than the competitors, and I believe that (somewhat paradoxically) this will result in an overall better, safer, and more reliable system, when it comes to the 20% long tail.
Lidar+vision will probably dominate in fair weather driving. I'm skeptical it's of much benefit in bad weather, and that long tail is the hard part.
Alternatively, no LIDAR means you aren't gathering corresponding LIDAR and vision data that you can use to improve detection of the vision system.
Right, because all those companies that use LIDAR are making billions driving people around. Oh, wait, actually they are burning 100 of millions every year.
> I think most hilarious is that both Karpathy and Musk claim the LIDAR systems are too expensive.
That's literally the opposite of what Musk says about Lidar. He LITERALLY said he wouldn't use them if they were free.
> yet stumbling towards autonomy on vision only is
Look up the Marginal Revolution from 1870.
> The entire ...
... The entire paragraph is an exercise in missing the point.
It seems really what you are saying is not that Tesla are charlatan but the whole industry is.
The entire industry has an air of charlatanism. Tesla is the biggest charlatan of the bunch. The LIDAR-using ventures at least have a snowball's chance in hell. Tesla has no chance with vision alone.
Well, that's a truth with modifications [1]
The whole vision vs. LIDAR stuff is a distraction as long as Tesla “AI” doesn’t have common sense.
It literally doesn’t know what it’s doing, and the tail of edge cases to "fit" the models is infinitely long. ANNs are fundamentally backwards looking and cannot adapt to unforeseen or even slightly unusual combination of circumstances. It will go fine for n miles and will dramatically fail at mile n+1 where a new situation requires understanding of ones surroundings, and n is arbitrary number.
It would be more honest to show the cases where it missed, thankfully there is no lack of them in “FSD beta“ videos on YouTube.
the key word is "etc" which expands into infinite tail, which no big data training on farms of GPUs would ever help with.
Humans and other animals have an ability to understand the scene and generalize from prior experiences to infinite set of new and unexpected circumstances, the "common sense" these dumb curve-fitting models are fundamentally lacking.
Assuming the car is reasonably good at this, it has the advantage that it can see in every direction at once.
I don't think self-driving cars will ever be perfect, but I think they will quickly become less-lethal than the average human.
Let’s see how well does a Tesla react to exceptional cases where actual decision making is required, few data sample is available, etc. Statistics can be misleading with rare events.
It's also worth pointing out that humans are notoriously bad at "exceptional cases" sometimes. People full-stop on the freeway for small animals to cross sometimes or panic when adjusting their eyes out of tunnels.
I do think it would be interesting to compare the two (computers vs humans) and look at the venn diagram of crash cases.
The human vision system operates at a bitrate equivalent of well beyond 500 gigabits per second. Resorting to "humans can do it with vision alone" is only a sufficient argument if your computer vision system can match that.
Do you have a source for this claim? Academic sources suggest numbers 4-5 orders of magnitude smaller: 6Mbps [1], 10Mbps [2].
[1] The Oxford Companion to the Mind (1987)
We know the eye has highly nonuniform resolution, with the high resolution region moving around all the time to cover areas of interest. I agree this system has something like 10 Mbps actual bitrate.
But we don't know how to make such a system for computer vision. Our CV systems all have uniform resolution. And thus a CV system needs to match the peak resolution of the human vision system. That requires a ~500 megapixel resolution at 24 bits per pixel and ~50 frames per second.
An AI driver doesn't necessarily need to match or exceed a human driver's sensors in order to drive as well or better. The human eye does indeed have very impressive resolution and tracking, but the ability to attend to a certain area of interest also means that you miss things in other areas.
Instead of using the human as the reference point, we can turn the question around and ask instead: given a video feed from a current generation Tesla, would a human be able to drive safely? I would suggest yes, though it would take some adjustment from existing driving habits. If a human could do it with Tesla's current sensors then an AI almost certainly can eventually do better with the same sensors.
I actually think the bigger issue with Tesla's approach (and I say this as a generally happy Tesla owner... though the timing is terrible since it had to be towed to the service center today) is their insistence on minimal maps/local data for driving. I understand why they want to do that for scaling purposes, but there are lots of intersections that are confusing for your average human the first time they come to it. I expect figuring out the traffic pattern in a confusing intersection is the harder AI problem.
And despite humans having legs and birds having articulated wings, ground vehicles use wheels and airplanes use fixed wings with propellers or jets. Why?
Just because biology solves a problem a certain way doesn't make that same approach optimal for machines. Typically with machines you need to work around other systemic deficiencies.
Humans may have only two cameras, but those cameras are connected to a vision, spatial understanding, and reasoning system that is light years beyond what current day AI is capable of. But the good news is that we can make up for AI deficiencies by giving vehicles super-human sensing ability which makes those problems easier.
Tesla's approach seems to be one in which they think the sensors are the expensive/hard part and solving general AI is the cheap/easy part. I think plenty of people would not agree.
Vision is only used to periodically adjust this model of the world - most of the time, we are living in our heads. This is so true that we normally don't even notice that we entirely lose our vision every time we blink or move our eyes, and we don't notice the extreme blurriness that we have in areas of our 'vision' where we are not currently focusing our eyes.
It's very educational to visit some areas built with non - common sense objects to understand just how limited human vision is if you remove our understanding of the world from it.
What a ridiculously useless term.
And humans also fail after some arbitrary number of m+1 miles. And in many cases far more catastrophically then AI.
https://mobile.twitter.com/TaylorOgan/status/140705191831739...
[1]: https://mobile.twitter.com/TaylorOgan/status/140709356765125...
> Purpose is recruiting.