Hacker News Reader

Comment by throwawaylinux | Hacker News Reader

Do lidar driving systems have objectively better performance? Genuine question (I don't keep up with driving "AI"). I'm skeptical if they would never kill anyone or cause any crashes.

That said, I see sign poles flattened all the time. Paint and tire marks all over barriers, walls, and islands. People crash all the time. The old anecdote game isn't really useful, automatic driving could be 100x better than human drivers in all measurable metrics and you'd still be able to upload dashcam videos of it fucking up.

View on HN

> Do lidar driving systems have objectively better performance?

Yes. CA DMV accident and disengagement data for self driving systems using Lidar show they are indeed better (and by miles).

Note that Tesla refuses to share FSD beta data because they classify it as a level 2 system. I'm willing to bet their disengagement rate is in the single digits. For comparison, Waymo and Cruise both had around 30,000 miles per disengagement in 2020 according to CA DMV records.

So you know that you don't have equivalent data for Tesla but that doesn't stop you from claiming that Waymo is better than Tesla? Based on data that neither you nor anyone else have?

There’s no evidence for Tesla FSD having a better safety record. When they used to report to the DMV, they were still nowhere near Waymo and Cruise. Since then, we all know how “capable” FSD really is. There are tons of videos showing FSD screwing up the most basic of scenarios. So it’s a very reasonable conclusion.

Almost all of the videos on Youtube show interventions every couple minutes at least, let's generously say 1 per 5 miles. Do you think off film they're doing 100k miles without intervention for each of those to make up for it?

Are you talking about entertainment videos where some random person tests there own car where the cut to the beginning the most difficult routes they have found where previous software and sometimes even human drivers have difficulty, with the uneventful parts often cut or sped up?

If so, I've seen quite a few of those for the Tesla FSD beta, but none for any other cars. Not that I try to find them, the Tesla hype must just make those come up more. Do you have any of them with other cars showing them go through these same routes without problems, that FSD struggles with? That might be fun to watch, even if it's not significant actual data.

> Yes. CA DMV accident and disengagement data for self driving systems using Lidar show they are indeed better (and by miles).

This seems to be a self reporting scheme for companies testing self driving cars in the state. I could find little or nothing about the details of the testing being done or methodology used by each company.

> Note that Tesla refuses to share FSD beta data because they classify it as a level 2 system. I'm willing to bet their disengagement rate is in the single digits.

Hypothetically, could company A use company employees as human drivers driven specific routes chosen by the company, while company B used volunteers from the public in their own cars driving their own routes, and both data sets would go into the same CA DMV data?

Could company A have any incentive to improve miles per disengagement to the point where they might intensively train their drivers, direct them about certain situations which don't require disengagement, etc., choose to drive on routes known to be less difficult? And could the drivers employed to do so much test driving rapidly learn more about the behavior and know when they can avoid disengagement?

If so, then I'm a bit skeptical about the objectivity of that data.

Given Tesla FSD numbers are not included, are you saying that data is showing lidar systems are better by showing numbers for e.g., Waymo and Cruise vs other non-Tesla companies which don't use lidar? Which companies are those? I'd like to understand how you're using this CA DMV data.

> This seems to be a self reporting scheme for companies testing self driving cars in the state. I could find little or nothing about the details of the testing being done or methodology used by each company.

It is a self reporting scheme. Waymo publishes their methodology at [1], Aurora at [2]. I'm not aware who else publishes their methodology, so it's up to you how much you trust them.

> Hypothetically, could company A use company employees as human drivers driven specific routes chosen by the company, while company B used volunteers from the public in their own cars driving their own routes, and both data sets would go into the same CA DMV data?

Yes, why shouldn't it go to the same dataset? Company A tests in an area defined by their Operational Design Domain (ODD), company B's ODD is simply everywhere in CA (that's what they've committed to).

> Could company A have any incentive to improve miles per disengagement to the point where they might intensively train their drivers, direct them about certain situations which don't require disengagement, etc., choose to drive on routes known to be less difficult? And could the drivers employed to do so much test driving rapidly learn more about the behavior and know when they can avoid disengagement?

Sure, they can drive only in routes that are less difficult. But they have a product to develop (a public robotaxi service) and areas to expand. So they don't really have an incentive to exclusively work towards artificially reducing disengagement rate.

> Given Tesla FSD numbers are not included, are you saying that data is showing lidar systems are better by showing numbers for e.g., Waymo and Cruise vs other non-Tesla companies which don't use lidar? Which companies are those? I'd like to understand how you're using this CA DMV data.

Nope, I indeed mean they are better than Tesla. There are no other (aspiring or current) L4+ systems that don't use lidar. When Tesla used to report disengagement rate in CA, they were still worse than their competitors. That and the tons of YouTube videos showing FSD being unable to handle basic scenarios tells us their current disengagement rate is far worse than competitors.

What they should be really doing is start being transparent and report safety data like everyone else.

[1] https://waymo.com/safety

[2] https://safetycaseframework.aurora.tech/gsn

> Yes, why shouldn't it go to the same dataset?

I didn't say it should or shouldn't, I was just asking about the data you are using.

> Sure, they can drive only in routes that are less difficult. But they have a product to develop (a public robotaxi service) and areas to expand. So they don't really have an incentive to exclusively work towards artificially reducing disengagement rate.

Oh they absolutely do if people buying their shares, their services, employing them, or writing legislation, or just general public attitudes are influenced by this data.

The marginal cost of cars is so low given the costs involved that it would be trivial to add easy miles while still doing difficult routes to develop the product. Including doing most of the difficult work in other states.

> Nope, I indeed mean they are better than Tesla. There are no other (aspiring or current) L4+ systems that don't use lidar. When Tesla used to report disengagement rate in CA, they were still worse than their competitors. That and the tons of YouTube videos showing FSD being unable to handle basic scenarios tells us their current disengagement rate is far worse than competitors.

Okay, so not the objective data I wanted but thanks for helping.

> What they should be really doing is start being transparent and report safety data like everyone else.

I don't think this is really a matter of transparency given all the problems with the data and tests we've just identified. When the products and are made available to the public that would be a different matter.

> Oh they absolutely do if people buying their shares, their services, employing them, or writing legislation, or just general public attitudes are influenced by this data.

None of this matters if they don't show product development. They have a market to serve and profitability to think of. There's only so much PR you gain out of artificially reducing disengagement rate. You can't seriously believe this is a real strategy employed by a dozen SDCs.

> I don't think this is really a matter of transparency given all the problems with the data and tests we've just identified. When the products and are made available to the public that would be a different matter.

Even if you think self reporting data is a problem, the solution is not to encourage companies like Tesla to continue to be opaque and not comply with rules. Since Tesla FSD is already in the public's hands (with more weekly rollouts planned), it is indeed a matter of transparency.

I'm not sure what any of this has to do with my asking about objective data, even with your very generous idea about the definition of objective data.

> Do lidar driving systems have objectively better performance?

In one sense yes: accurate ranging is an inherent capability of a lidar system that a passive optical system needs additional algorithms to achieve (e.g., simultaneous localization and mapping (SLAM)).

In an active system like lidar you know when you’ve transmitted energy and you can measure the delay until you receive an attenuated version of it scattered from objects in the environment. And you know the velocity of the energy so you can solve for distance.

You don't see the difference between the one guy willfully crashing his Airbus into a mountain and Boeing having MCAS that makes the control planes not work?

The big fucking difference is that the latter was a system installed into all of them! The failure mode was systematic! A different car with the same software facing a similar situation would go straight for the pole again!

I do. I don't see how your comment addresses what I wrote though.

Reply on news.ycombinator.com