"Yeah, we don't need Lidar at all!" except when it's going to kill you.
That said, I see sign poles flattened all the time. Paint and tire marks all over barriers, walls, and islands. People crash all the time. The old anecdote game isn't really useful, automatic driving could be 100x better than human drivers in all measurable metrics and you'd still be able to upload dashcam videos of it fucking up.
The big fucking difference is that the latter was a system installed into all of them! The failure mode was systematic! A different car with the same software facing a similar situation would go straight for the pole again!
Yes. CA DMV accident and disengagement data for self driving systems using Lidar show they are indeed better (and by miles).
Note that Tesla refuses to share FSD beta data because they classify it as a level 2 system. I'm willing to bet their disengagement rate is in the single digits. For comparison, Waymo and Cruise both had around 30,000 miles per disengagement in 2020 according to CA DMV records.
If so, I've seen quite a few of those for the Tesla FSD beta, but none for any other cars. Not that I try to find them, the Tesla hype must just make those come up more. Do you have any of them with other cars showing them go through these same routes without problems, that FSD struggles with? That might be fun to watch, even if it's not significant actual data.
This seems to be a self reporting scheme for companies testing self driving cars in the state. I could find little or nothing about the details of the testing being done or methodology used by each company.
> Note that Tesla refuses to share FSD beta data because they classify it as a level 2 system. I'm willing to bet their disengagement rate is in the single digits.
Hypothetically, could company A use company employees as human drivers driven specific routes chosen by the company, while company B used volunteers from the public in their own cars driving their own routes, and both data sets would go into the same CA DMV data?
Could company A have any incentive to improve miles per disengagement to the point where they might intensively train their drivers, direct them about certain situations which don't require disengagement, etc., choose to drive on routes known to be less difficult? And could the drivers employed to do so much test driving rapidly learn more about the behavior and know when they can avoid disengagement?
If so, then I'm a bit skeptical about the objectivity of that data.
Given Tesla FSD numbers are not included, are you saying that data is showing lidar systems are better by showing numbers for e.g., Waymo and Cruise vs other non-Tesla companies which don't use lidar? Which companies are those? I'd like to understand how you're using this CA DMV data.
It is a self reporting scheme. Waymo publishes their methodology at [1], Aurora at [2]. I'm not aware who else publishes their methodology, so it's up to you how much you trust them.
> Hypothetically, could company A use company employees as human drivers driven specific routes chosen by the company, while company B used volunteers from the public in their own cars driving their own routes, and both data sets would go into the same CA DMV data?
Yes, why shouldn't it go to the same dataset? Company A tests in an area defined by their Operational Design Domain (ODD), company B's ODD is simply everywhere in CA (that's what they've committed to).
> Could company A have any incentive to improve miles per disengagement to the point where they might intensively train their drivers, direct them about certain situations which don't require disengagement, etc., choose to drive on routes known to be less difficult? And could the drivers employed to do so much test driving rapidly learn more about the behavior and know when they can avoid disengagement?
Sure, they can drive only in routes that are less difficult. But they have a product to develop (a public robotaxi service) and areas to expand. So they don't really have an incentive to exclusively work towards artificially reducing disengagement rate.
> Given Tesla FSD numbers are not included, are you saying that data is showing lidar systems are better by showing numbers for e.g., Waymo and Cruise vs other non-Tesla companies which don't use lidar? Which companies are those? I'd like to understand how you're using this CA DMV data.
Nope, I indeed mean they are better than Tesla. There are no other (aspiring or current) L4+ systems that don't use lidar. When Tesla used to report disengagement rate in CA, they were still worse than their competitors. That and the tons of YouTube videos showing FSD being unable to handle basic scenarios tells us their current disengagement rate is far worse than competitors.
What they should be really doing is start being transparent and report safety data like everyone else.
I didn't say it should or shouldn't, I was just asking about the data you are using.
> Sure, they can drive only in routes that are less difficult. But they have a product to develop (a public robotaxi service) and areas to expand. So they don't really have an incentive to exclusively work towards artificially reducing disengagement rate.
Oh they absolutely do if people buying their shares, their services, employing them, or writing legislation, or just general public attitudes are influenced by this data.
The marginal cost of cars is so low given the costs involved that it would be trivial to add easy miles while still doing difficult routes to develop the product. Including doing most of the difficult work in other states.
> Nope, I indeed mean they are better than Tesla. There are no other (aspiring or current) L4+ systems that don't use lidar. When Tesla used to report disengagement rate in CA, they were still worse than their competitors. That and the tons of YouTube videos showing FSD being unable to handle basic scenarios tells us their current disengagement rate is far worse than competitors.
Okay, so not the objective data I wanted but thanks for helping.
> What they should be really doing is start being transparent and report safety data like everyone else.
I don't think this is really a matter of transparency given all the problems with the data and tests we've just identified. When the products and are made available to the public that would be a different matter.
None of this matters if they don't show product development. They have a market to serve and profitability to think of. There's only so much PR you gain out of artificially reducing disengagement rate. You can't seriously believe this is a real strategy employed by a dozen SDCs.
> I don't think this is really a matter of transparency given all the problems with the data and tests we've just identified. When the products and are made available to the public that would be a different matter.
Even if you think self reporting data is a problem, the solution is not to encourage companies like Tesla to continue to be opaque and not comply with rules. Since Tesla FSD is already in the public's hands (with more weekly rollouts planned), it is indeed a matter of transparency.
In one sense yes: accurate ranging is an inherent capability of a lidar system that a passive optical system needs additional algorithms to achieve (e.g., simultaneous localization and mapping (SLAM)).
In an active system like lidar you know when you’ve transmitted energy and you can measure the delay until you receive an attenuated version of it scattered from objects in the environment. And you know the velocity of the energy so you can solve for distance.
(The best theory I've seen presented for this particular bug, FWIW, is that it was the shadows and training data at fault. Teslas can, obviously, recognize street lights. But this one evaded detection at a distance because it was inside the dark shadow and presumably the contract detection failed to resolve it. And once it was in the near field, the recognizers don't have as much training data and failed to see it.)
This was probably true in 2016, but not anymore. Waymo’s 5th gen lidar and Argo AI’s lidar all show massive performance and resolution improvements. They have been game changers. Anyone not using lidar in 2021 is just handicapping themselves, as we are seeing with Tesla FSD performance.
As far as I know the specs for Waymo's 5th generation lidar are not publicly available. Can you link to them?
How much work did you do with Waymo's 5th generation lidar? Or any lidar for that matter?
Do you have any qualifications to claim that a lidar that isn't even available to you is capable of doing a very specific task?
If you want to hear from an expert (which I am not), here is Brad Templeton talking about Waymo and Argo's lidar improvements [3].
[1] https://www.youtube.com/watch?v=COgEQuqTAug&t=11601s
[2] https://www.youtube.com/watch?v=I_tYxMh3ddA
[3] https://www.forbes.com/sites/bradtempleton/2021/10/11/new-li...