Hacker News Reader

This is not good enough to handle edge cases.

It's a little bit like ChatGPT: ChatGPT is a next-word prediction engine. It has ingested and analyzed "human data" which has given it a take on what might come after a prompt. With good training data, a large enough token window and a helping of RLHF this can do very cool things. But there's no actual reasoning going on.

An "edge case" is a prompt that throws this approach for a spin, because statistics over the training data corpus don't yield a good prediction on it. For example because it's novel or its structure is uncommon. In those cases ChatGPT will do its best and confidently return something very dumb.

With the driving problem, Tesla and others are trying to mitigate this with two approaches:

- superstructures of rule-based planning to keep the predictions within safe lanes (no pun intended), i.e. bake in some reasoning after all

- trying to write simulations that auto-generate "edge cases" or just stuff that is underrepresented in the training data set and add those simulated 3d renders to editorialize the predictions

It's currently still unclear if this is enough to solve the problem or if we're still some innovations (e.g. ones that get us closer to AGI) away from being able to make it work.

I wonder if self-driving doesn't actually require reasoning, though. Most of us are able to drive instinctively, without fully engaging all of our attention; we certainly aren't solving problems with deep analytical thinking. Perhaps it's a problem that's well-suited to the approach an AI like ChatGPT takes?

I agree. I go weeks at a time without having to do any critical thinking whatsoever.

There probably are some situations that do actually require a looping reasoning process that a simple feed-forward neural network can't do, but they are fairly rare.

The most common ones probably actually involve complex parking lot navigation tasks, where you have to negotiate the space with other drivers, understand their intentions, yield space, take space, etc.

You may end up using reasoning more often than you think. I run into situations that require my full attention and are very likely un-AI-able once a week or so. (And I live in California.)

> Most of us are able to drive instinctively, without fully engaging all of our attention; we certainly aren't solving problems with deep analytical thinking

Depends. Many of the type of situations where FSD runs into trouble or disengages are also the ones where a human would slow down/pause and ponder a moment what to do next. In the FSD case, it then often eventually gives up and disengages, because it can't confidently predict what action to take. Humans show better performance in those cases.

If you ask me what separates a human from ChatGPT right now, it's that ability to use reasoning to fill in for bad training data. If, as some people have posited, the training data for ChatGPT-like systems is possibly already exhausted, we need some new ideas.

But humans are good at that. :-)

Aside from that, note also that your learned instinctive predictions take into account many more parameters than the models a Tesla currently runs anyway. For example you have an understanding of social dynamics (so what those other human drivers think and what might be acting on them, etc.) that is far in excess of what the Tesla models can learn implicitly from videos that observe them driving. With some of these things we have no idea yet how to feed them into the models.

>This is not good enough to handle edge cases.

Is Tesla doing any better, though?

At least comma can handle the "huge unmovable object in front of me, I should brake" edge case.

OP claimed that Comma's approach would be able to handle edge cases better, but I don't think it's fundamentally a solution to that problem. Tesla is struggling with the same tech limitations certainly.

I think it will handle edges better because it's explicitly not trying to classify everything into objects (for the purpose of loading into their virtual representation).

So it doesn't need to "understand" a situation in terms of what is physically true, it just needs to understand what information is valuable for the task of driving.

Consider this edge case: https://www.youtube.com/shorts/kM-xBgz26pE

A utility truck carrying a load of new traffic lights. Tesla correctly identifies them as traffic lights and loads virtual representations into its world model. It doesn't seem to cause any weird behavior (thankfully), but this demonstrates a case where doing this fancy classification actually makes your system less robust.

I'd be interested in a direct comparison of how Comma's models handle the same situation. Has someone tried?

I know, and I'm saying that whatever Tesla is doing, along with those "mitigations" you mentioned, is crap because of the several crashes where it has failed to detect big stationary objects in front of the car.

I haven't seen comma doing stuff as dumb as that, so I'm also betting on comma's approach.

I have no stake in the Comma vs. Tesla, I just don't think the Comma approach can get us all the way to a full solution either.

What I think is interesting is that if you assume that there's still a couple of new ideas left before the whole thing can work, it matters who is the most poised to adopt (or even to have) the new ideas and turn them around into a product.

I also don't know who that will be. A vertically integrated company like Tesla? A tech supplier like Waymo? A nimble startup like Comma?

As a Comma owner (2 Commas in fact), I feel like Comma's main innovation is only releasing features (on the main branch at least) that they feel are quite battle tested, and at the same time not being afraid to lean into the fact that Comma is somewhat limited and isn't full FSD.

My Commas are bulletproof at doing what I expect them to do, and they don't do what I don't expect them to do. And from my perspective as a driver, it's incredibly, incredibly useful. When driving is easy and mundane I can trust that Comma can handle it (and Comma will tell me if it can't), and when driving is hard then I don't expect Comma to do handle every complex situation (though it does do a good job most of the time). They don't overpromise, and they deliver what they promise.

I'm not convinced this method will get to FSB, but I from my perspective I don't think that is necessary. I genuinely believe if every car on the road only had Comma's capabilities and no more, that would on its own reduce highway accidents by like 50%. I really wish some car manufacturers would lean into this and work with Comma on making a first-hand integration. I would 100% buy any type of car from any manufacturer if it had first-hand Comma support.

>>takes in the last few frames of video

isn't that how uber killed a woman walking a bike in phoenix? During the critical last second(s) the model didn't know what she was and kind of ignored her even though previous seconds had tagged her as a moving cyclist.

From the wiki article about the Uber incident:

> The recorded telemetry showed the system had detected Herzberg six seconds before the crash, and classified her first as an unknown object, then as a vehicle, and finally as a bicycle, each of which had a different predicted path according to the autonomy logic. 1.3 seconds prior to the impact, the system determined that emergency braking was required, which is normally performed by the vehicle operator. However, the system was not designed to alert the operator, and did not make an emergency stop on its own accord, as "emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior", according to NTSB

So Uber had a lot of systemtic issues leading to the crash, but even if most of those were fixed, I think Comma's approach works better here because it is explicitly not trying to do any of this fancy classification and prediction. If a human sees an unknown moving object near the road at night, they start slowing down even before they have identified the object. Comma is trained to just do what a human would do, so it will also slow down.

[1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg

Jesus that's right, I forgot they had turned off braking, fair enough to the Comma folks.

> takes in the last few frames of video, and outputs steering + accelerator + brake commands

Isn't this pretty close to what Tesla does?

Cars phone home with timestamped video from multiple cameras plus data on what hundreds of thousands of human drivers do, also timestamped.

Use all the human events to train a DNN to make similar decisions as humans.

Then augment that with GPS, maps and routing to a destination.

No, Tesla does something much, much more complex. They build an entire 3D representation of the world and do path planning inside that virtual model. Check out the videos from the last Tesla AI day. They are creating a voxel representation of the world, doing object classification and loading objects into the world, doing monte carlo tree search path planning in the virtual world, etc.

I'm not sure where you heard this, but you heard wrong. Tesla absolutely does use real world data to train their neural nets.

I think you have misinterpreted my comment.

Of course Tesla uses real world data. My claim is that their self driving system it not end-to-end. That is, it's not "just" a neural network that takes sensor data as input and gives steering/accelerator/brake commands as output.

They take in data from all their sensors, fuse it together, and create a real-time virtual copy of the world. They do planning inside this virtual copy of the real world. This has been demonstrated in many Tesla presentations. The FSD visualization on the screen of the Tesla is itself a depiction of the virtual world being constructed from the sensor data.