Waymo AV Freaked Out by Traffic Cones, Blocks Traffic, Evades Support Vehicles
jalopnik.com
jalopnik.com
This is the kind of situation that confuses new human drivers too. Interpreting it basically requires that you have a model in your head of what cones are used for and the ability to guess that the traffic approaching from the left in only one lane indicates that the traffic to the right must use the left lane.
I'd be very curious if this conework was even legal. If you want to guide left-turning vehicles around the right lane you need to put something IN FRONT of the lane you closed. And that applies to side streets too, I'm almost sure.
Watch /r/idiotsincars for a bit, you'll see people misidentify this kind of situation all the time.
Frankly what the car did, stopping and calling for help, is the right move. A new human driver would almost certainly have just pulled out into the wrong lane.
Honestly, if it had just stayed on the side street, I wouldn't be so critical, but the fact that even after an assistance team was dispatched, the car pulled into the main highway and then immediately stopped would have frightened me enough to get out of the car and wait on the sidewalk. You can also see a moment where the car starts to pull out into traffic as another car is passing it and then immediately slams on the brakes.
Stop it. Waymo vehicles (Tesla autonomy too, FWIW) are extremely safe, that is just an experimental fact. The circumstance in question here has nothing to do with safety, really. The bug is that it caused a traffic obstruction while the (human) operators attempted to get the car oriented.
Hyperbolizing the circumstances is really not appropriate. This was not a safety situation, really, at any point.
There's a joke among cyclists: "If you want to get away with killing someone just be sure that you're driving a car and they're riding a bike." We could make a similar joke about these killer robot cars. "If you want to test killer robots on the public just make sure they are shaped like cars." (And if you need to, you can throw in a "safety driver" as a sort of moral crumple zone to deflect the blame when your robot kills e.g. Elaine Herzberg. https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg#After... )
- - - -
We could build self-driving golf carts (slow, light, not so capable of killing) and work up to cars, eh?
We don't have to go straight to fast heavy cars, eh?
I acknowledge that it is weird that the remote assistance team can't remotely stop the vehicle and I agree that it should have behaved differently but maybe there is some kind of engineering reason or just a glitch. It seems to recalculate what to do three times.
It was incredible to watch how Waymo was able to plot the huge number of cars accurately around the vehicle throughout the drive. But as soon as something creative was required, it all fell apart.
I still remember a talk where someone from the original DARPA challenge was recounting an episode where two autonomous cars arrived at an intersection precisely at the same time. They deadlocked because they were waiting for each other to go. A programmer added jitter to work around that situation, but it is one of those things humans innately would figure out.
I have a hunch that 90% of self-driving could be solved like that, but then there's always that 10% of situations where there's nobody in front or someone who doesn't know what they're doing.
But yes it would be a good heuristic.
The thing is, the strategy of 'follow other traffic' needs exit rules, and while it is relatively straightforward to begin the strategy when stuck, deciding when to abandon it is less clear-cut. Did the left-turning vehicle do so because this was the correct way to deal with the problem, or did it have a different goal?
"He had a tremendous propensity for getting lost when driving. This was largely because of his method of “Zen” navigation, which was simply to find any car that looked as if it knew where it was going and follow it. The results were more often surprising than successful, but he felt it was worth it for the sake of the few occasions when it was both."
A random sleep to avoid a race condition.
If a bunch of randoms on the Internet can think of using the other vehicles as guides for the right way to go, why hasn't Waymo?
Also, why don't the roadside assistance people or the woman on the phone not have an off button to stop their cars randomly taking off on them? Other machinery has an e-stop, why not this?
Tesla has since improved recognition to prevent that situation, but the point remains: blindly following other vehicles is a common mistake. Remember all the "Do Not Follow" signs on construction vehicles?
4:30 look at the cars coming from the left and turning right (down the screen). Google hallucinates two cars right there, and thats just the start of the half hour journey.
Google are probably planning for 100% of the cars on the road being AI-controlled, at which point it no longer makes sense to copy other car’s behaviour, because they are all AIs.
Seeing how monopolistic Google is in other areas, it would not surprise me if they are chasing a self-driving monopoly, too.
You cannot drive into that lane without hitting those cones. It's hard to see from the camera angle in the backseat but a driver in the front would definitely understand that the lane is blocked.
I wonder if the traffic on the left lane somehow shifted the code/network into favoring that it was only the right lane that was closed. That may have gotten it moving after the first hang up.
However, once it turned into the left lane and passed the first cone (which was offset to the right of the divider), it saw the second cone placed in the center and forgot that it was the right lane that was closed.
I think it pulled over because it thought that both lanes were closed or did not know which one was closed. It’s modeling of a sequence of cones closing lanes and retention of that model seems inadequate.
I'd be very curious if this conework was even legal
Here in California, it happens all the time. There was a half block of EL Camino Real in Sunnyvale this afternoon that was simply coned off with no signage and no warning by construction people working on a new building. I called it in to Sunnyvale DPS, and they were totally unaware of it. There were two near-collisions in the 45 seconds I was within view. You have day laborers with no clue about traffic safety and ignorant of context given hand signs and no instructions, let alone legal standards.
Even if you place cones and signage perfectly per code, you still have to deal with signs or cones being knocked out of position or out of view.
P.S. I hope similar tech will be built for other types of road use, so cars alone don't shape our roads even more.
Seems like if those confused drivers had merely stopped where they were, safety would have been improved!
Again, the question isn't whether or not automation technology can be confused by bad signage, we know it can. Nor is there a question that humans can be confused by the same situation. The question is, when faced with situations like this, we can trust the automation to operate safely.
And it seems like Waymo passed the test.
It's true the externalities are different for the different technologies though. If you put down bad cones currently, you raise the likelihood of an accident. If you mark your lane badly in the era of automated vehicles, you cause a traffic jam. I know which I'd prefer.
It did not seem so to me. Blocking the road (when it was clearly possible to get out of the way) increased the danger to others through inaction.
Issac Asimov would tut tut.
The choice was between block the road, or pull off to the side, when it was clearly safe to do so.
The car did pull off to the side. Those cones mark the side of the road. What you're demanding it do is to cross the cones to pull off the closed-off shoulder of the road. Which is (1) illegal and (2) only safe because of assumptions you're making about why that road was closed.
I mean, consider that they had a giant automated Waymo-branded repaving machine coming down in that lane. Would pulling off the the side have been the right choice?
Again, it's true that an AI isn't going to have the same context that you do as a functioning human driver. And so it's going to make different decisions in pursuit of safety than you would. That's not evidence that it's unsafe. This behavior simply wasn't unsafe.
That's not what I saw.
The car attempted to follow it's planned route to use the right lane, and stopped in place when it saw the next traffic cone, partially blocking the left lane.
Then it backed up and fully blocked the left lane.
Because making AVs work to the last step, to account for every possibility, we're not there and we're not getting there for a long long long time, so have then the bunch of AV-manufacturers come together and standardize on a secure (lol) communication protocol and give it the crutch that it apparently desperately needs.
FWIW: it really doesn't seem like it has to be that complicated. The car could see the cones just find and knew not to cross them. They just needed 4-5x more cones than they used to make it clear to the car[1] where the lane was actually going.
[1] And human drivers! Again, this conework was just plain awful, period. I'm 100% convinced that the poor Waymo device wasn't the only driver making bad decisions in that intersection that day.
The important point is that a human driver (as all the human drivers in the video) has no trouble. A lot of driving is not by the book, but humans coordinate and go with the flow. The software in these cars breaks down as soon as anything unexpected happens.
In general, I would imagine that following a route plan to the lane level of detail is important; I can see that it would be when approaching a turn, for example. Also, when the car first encounters the cones, it only recognizes that the lane is blocked for a short distance: after that, each cone encountered seems to be a separate problem. Humans easily generalize a line of cones to a blocked lane and modify their plans accordingly, but we do not have AGI yet.
https://youtu.be/zdKCQKBvH-A?t=1818
They way I understand that is the route planner, which I assume is ran completely separately to the self-driving car, removes un-navigable spaces from the route planning. So in this situation, the roadside assistance driver is surprised that the entire lane has not been removed from the route planner.
That's pretty damning and it makes me wonder if there is a huge gap in autonomy here that is being papered over by clever route planning (ie. avoid that lane altogether and the AI won't have to worry about the cones).
The AI should be able to figure out on its own that the right lane was closed, it shouldn't need route planning to do that for it.
Take the moment when the car first gets stuck. It attempts a right turn, spots the cones just fine, and correctly makes the assessment that it can avoid the cones and proceed in the right lane after the turn.
As a human, our thinking process does not stop there. We notice that the traffic already flowing on the street is _not_ using the right hand lane. We notice that the cones are vaguely in a merge pattern to indicate that cars entering from the side street should merge left - if you were looking at the cones from a vantage point behind where the cones are placed, and if you can call the first two cones enough of a "please merge left" pattern.
The Waymo car can't see that, though. It's located sort of off to one side, attempting to reason about it from a high angle. And two cones from this angle could be a lot of things, and it's not 100% clear the cones close off the right lane.
One theory might be that the Waymo model has been trained on a lot of situations but not this specific situation - something about the high angle, something about the position of the cones, something about the route plan managed to find a loophole in the AI training set and produce a weird result.
That's just the limitations of the current state-of-the-art Machine Learning. Find a glaring incorrect answer? Retrain with more training data.
Part of what makes full Artificial General Intelligence (AGI) with deductive reasoning can retrain itself without making mistakes along the way, unless something incomprehensible happens. And I use the word "incomprehensible" here to point straight at AGI's ability to _comprehend_ the situation at a much higher level than current machine learning techniques.
tl;dr: The Waymo Car didn't get it. And that's about what I would expect.
The real gem is that the _humans_ in the assistance team were fighting each other. The remote control operator kept jamming the "unstick yourself" button even as the roadside assistance guys were walking up to take over. The real danger, I think, is more of this in the future: the remote operators defeating all the safeties. Since the remote operator is not at any real risk of personal harm, what do they care?
IMO it's more concerning that the vehicle keeps trying after it gets into a "call roadside assistance" mode rather than staying still or allowing the remote operator to assume manual control (say to move the vehicle out of the lane).
Also, it really seemed like the Rider Support agent didn't have complete information on the status of the car or any apparent ability to intervene. I would love to know about how incident response is organized behind the scenes; there wasn't a lot of communication to the occupant about what was happening, which isn't ideal. The Roadside assistance request apparently being dropped when the vehicle began moving again, but before it demonstrated it was able to navigate the conditions is a bit concerning as well. It makes the response feel disorganized, although again, it could just be lack of information passing to the occupant.
“That's pretty damning and it makes me wonder if there is a huge gap in autonomy here that is being papered over by clever route planning”
To which I said that clever route planning is not a hack but the only reasonable way forward.
I was just trying to say it in even more plain terms - because this whole thread was full of people who seemingly have no idea about how AVs are being approached today and are, as is usual for HN, suggesting they are smarter than the actual people working on the technology.
Intelligent route planning to compensate for the deficits of our current perception tech stack is a good thing and allows greater reliability.
This is what Waymo is doing and has done from the start. But this approach itself has significant limitations as the video shows. You have to automate the route planning process also, remember.
It has limitations but there is no evidence of there being a better alternative. Automating the route planning process is easier than behavior planning, path planning or perception.
Even as a human driver the conditions seem kind of weird. There's some cones placed on the dotted lines between two lanes but no obvious obstructions in either of the lanes. The vision system likely did not know which side to be on. Combine that with a truck driving by and snatching up some of the cones at one point as it was going along, and it is easy to see why the AV behaves erratically.
we really need to collectively get past this idea that safety is bounded on one side, across all facets of our lives, because it’s usually not and that induced complacency has a way of coming back to bite us. (that’s not to say that the various risks are always substantial, just that they’re not all sitting neatly on one side.)
As soon as the assistance team was dispatched, the car should have stayed where it was. Instead, it pulled out into a much busier road and stopped again.
But that isn't what happened (or would typically happen) in this type of situation. Usually the car blocking the lane will be enough of an impediment that a queue of traffic will build up behind to overtake it. Cars at the back of the queue will be coming to a stop more gradually.
Human drivers should not have critical failures just because there is an unexpected queue of traffic on the road.
This sort of thing happens all the time. And because of modern crumple zones and safety equipment, people have an excellent chance of living through it and doing it again.
You would not be able to drive into that lane without hitting cones straight-on until after the stall point.
> "...the Fleet Response team provided incorrect guidance..."
It sounds like the car automatically pulled over when it detected the cones (as intended), but the remote support team erroneously told it to forge ahead anyway, which resulted in the escalating series of problems. Not sure what the best solution for that particular issue would be -- better training for the remote support teams, I guess?
You can watch the video. It pulls over incorrectly at first and is blocking traffic, then the support team made a mistake, and a hilarious escalation ensues with the vehicle actually attempting to run away from the support team multiple times!
They deliberately can't drive the car remotely. If Waymo wants a human to drive the car, they send the human to where the car is, the human gets inside and drives it like any other car.
One easy thing that can go wrong, just as with any other human activity, is the humans make a mistake and their "clarification" is actively unhelpful. If the Waymo Driver is confused about which lane is coned off, and you "help" by telling it the wrong lane, unsurprisingly things will not be improved.
The best solution is always that the Waymo Driver learns how to do better on its own, if you imagine it sees this situation and just confidently picks the correct lane and uses it, Fleet Response never interacts at all, there is no opportunity for them to "provide incorrect guidance". So a bunch of the invisible effort at Waymo is on making the Waymo Driver just get stuff right on its own.
Letting the passenger drive would unfortunately involve the risk of exiting the vehicle in traffic, which the lawyers probably vetoed, even though it might be safer in some situations.
At any rate, the remote team in this case IS able to control the car, hence it trying to drive into traffic cones and Google blaming the humans. It appears that their interface just can't be very good, especially as it requires people to physically turn up in a truck - you definitely won't have good situational awareness if you're trying to drive a car through a web app whilst looking out the window at the other side of the road.
The remote assistant never does anything to affect the car onboard system. The only instance is when the assist car turns up and the car is put into hold mode for the driver to enter.
Yes, VR is far from perfect, granted. But is it worse than what they appear to be using instead? Apparently they couldn't see the traffic cones properly from whatever system they were using and repeatedly gave it bad instructions, even though what to do was obvious to the person in the car. It seems like VR should be compared to the existing system rather than theoretically perfect vision.
Nope. This was already explained once in this thread, and you ignored it. Here's what Google's statement actually says
> ... the Waymo Driver detected an unusual situation and requested the attention of a remote Fleet Response specialist to provide additional information. During that interaction, the Fleet Response team provided incorrect guidance, which made it challenging for the Waymo Driver to resume its intended route, and required Waymo's Roadside Assistance team to complete the trip.
They can remotely tell a Waymo, stuff like "this lane is closed" or "No, that's not a school bus, it's just a truck" - and it will add that to the information it has, this might enable it to complete the journey. If you phone your dad, asking what a street sign you saw means, and he tells you - is he now driving your car? No, clearly you are still driving the car, he just answered a question. If he says the symbol means "Free on-street parking" but it actually meant "No entry" you're going to get some problems if you try to act on this bad information.
I mean ultimately, you can try to justify this as a feature not a bug, but I would now feel far less safe getting into a Waymo knowing that they have such a poor level of control over their cars. You shouldn't find yourself stalled in the middle of a multi-line highway like that, that's just dangerous, especially when continuing and pulling over is something any human driver could do in seconds. Being limited to ad-hoc map patching is just not sufficient - there's a whole universe of bugs they might not be able to resolve in that way.
Not sure if Waymo is reading Waze info, those kind of alerts could have been updated there (for roadworks, etc)
Never letting my guard down with such a feature again!
taking all the best parking spots
There are a ton of situations where humans fail at driving too. Like white out snow or heavy rain and fog. Or confusing cones. :)
The local traffic engineering folks around where I live are gung-ho on traffic circles, but I've yet to see one in the US where a traffic signal wouldn't have been a better investment.
Me too; but traffic signals are more than capable of detecting empty streets and oncoming traffic and changing lights to green before you get to the intersection.
> They are better and safer in every way and stats back that up (not to mention better for the environment and your car maintenance bills)
Traffic circles are tolerable with low traffic, but when traffic increases, especially if the traffic is asymmetric, it's possible for some directions to be blocked for a long time, and there's no fairness mechanism. Because speeds in traffic circles are slower than in signaled intersections with a green, it takes longer for the backlog to clear the intersection. And, while there's still throughput when one entrance dominates and blocks other entrances, if one exit is blocked, it can block the whole intersection and there's zero throughput; in a signaled intersection, other exits can usually still be used even if one is jammed.
Slowing for a roundabout when I wouldn't need to slow for a traffic signal that would likely turn green for me is bad for the environment. Extra miles driven from longer paths, and wear on tires from taking a circular path instead of a straight path is bad for the environment and my maintenance bills. They use more space than a signaled intersection, and may displace trees or impact waterways and drainage.
It's hard to look for pedestrians crossing from the right while also looking for a space in traffic from the left. Navigating them on bicycles can be tricky too, as there are a lot of crossovers.
"Wear in tired from taking a circular path" -- is this supposed to be satire?
It may not be 'a problem', but it does add delay, and it ruins the narrative of better in all conditions. And it happens fairly frequently at the circle I transit most often. There's a car ferry, and when people drive off of that and eventually through this circle, there are directions they don't want to go, and if you're coming from one of those, you just have to sit and wait. And you wait much longer than a light would have you wait, and it takes the people coming from the ferry longer too.
> "Wear in tired from taking a circular path" -- is this supposed to be satire?
It's minimal, but so is the brake wear people point out as a benefit of doing something better than a 4-way stop.
Slowing down and making a slight turn does not compare to the fuel and maintenance cost coming to a full stop and getting the cars mass moving again.
And regardless of it being “hard” to see pedestrians the statistics back up them being far safer: you have to be an active participant and look rather then “green means go” and not pay attention.
I suggest watching that video I linked. I’ve driven in a number of countries and the ones with more traffic circles were always more enjoyable + easier as well as subjectivity had better drivers
Approximately every four way stop in the US could be replaced with a traffic circle and substantially increase throughput and reduce waste (brake wear for unnecessary stop, fuel consumption for accelerating away from the unnecessary stop).
Here in CA (at least silicon valley) approximately all traffic signals have sensors (it always catches me by surprise when I drive in other states and find there's no sensor so I have to wait endlessly for a green light).
But sensors are nowhere near a substitute for the efficiency of a traffic circle. With a traffic light with sensors you still have to stop for a short time, so it has all the disadvantages of a stop sign and none of the benefits of a traffic circle.
If you only have sensors at the line, then yeah, you need to stop.
As cars near a circle they need to slow down, follow the rules and allow some error. Being more cautious prevents more crashes and allow for more flow.
Automatic signals can be wrong, and promote reckless behaviour, confusion, less flow and more stops.
UK is all circles, with most of Europe following.
This is why chaotic traffic situations without lane markings or signals can still move fluidly without crashes. Autonomous vehicles would be at a standstill in that situation.
It's going to be interesting to me how you work out: "It's legally required to use the left turn lane, but since it's not plowed out well enough everyone is turning from the travel lane instead and this is temporarily acceptable to both other drivers and any law enforcement"
I can only imagine the chaos ensuing if another one or two waymo taxis would have arrived at the same location. Would they have gotten stuck behind each other. Simultaneously trying to desperately call for help and go around each other?
Since billions of people are driving unassisted every day, that's clearly not true.
And besides that. If such decision actually cause injury eventually, who should response for that? The AI designer? The car company? The AI itself? The car owner? We still don't have a definitive answer for that yet.
You could have a team of people sitting in HQ in a VR enviroment to take over more basic problems like this and there would be little delay. Then physical support for more challenging times.
A system like this could also be super handy for delivery type companies where the autopilot drives to the destination and the virtual driver takes over for the loading dock/driveway etc where human problem solving might help for the first few times a location is encountered.
And in case of disabled vehicles, accident cleanup, or other unplanned lane closures, sometimes the cones are just set out and you don't know anything about it in advance.
This is an OSHA violation by the construction workers. "When redirection of the road users’ normal path is required, they shall be channelized from the normal path to a new path."[1] There's supposed to be a diagonal line of cones across the closed lane to channel traffic away from the closed lane. See the drawings in [1].
Stopping was a good move.
[1] https://www.osha.gov/sites/default/files/2018-12/fy10_sh-210...
It caused other drivers to make illegal moves to get around, this just isn't true.
Even if it was, no other driver in the video had any trouble driving there. Self-driving cars, if it is ever to be viable, need to adjust to the world as it is.
[1] https://www.facebook.com/permalink.php?story_fbid=4133828946...
This setup wasn't completely MUTCD compliant, but, there are also some states with their own, state-specific editions of the MUTCD.
(This is why the trolley problem for AVs isn't real.)
Example running over an animal to avoid an accident, getting rear ended to avoid hitting a pedestrian, taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead.
It's going to make a decision between different risk factors and between different known costs.
Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch.
If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian?
What if you can slow to 20 and then ditch?
These situations won't hopefully be prevalent because a massive portion of our current issues are down to inattention or stupidity and we believe ai will eventually be much better at avoiding such problems but they are still important and interesting.
Yeah but not trolley problem style. "Don't swerve" is a valid strategy for a car.
> Example running over an animal to avoid an accident
Depends on the kind of accident you have in mind, so I can't give real feedback here.
> getting rear ended to avoid hitting a pedestrian
Always brake to avoid hitting a pedestrian. You don't choose whether to get rear-ended and it won't be your fault either.
> taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead.
When one of the choices is "sit there and wait longer", that's definitely not a trolley problem. It either thinks going forward is sufficiently safe or it doesn't. It can wait all day for someone to intervene.
> Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch.
If it's programmed not to swerve, it will hit the pedestrian, which is an acceptable outcome of an unacceptable situation.
> If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian?
It's sufficiently ok when the only alternative is so reckless. Getting into that situation is the not-ok part.
> What if you can slow to 20 and then ditch?
If you can slow to 20 then hit the pedestrian at 20.
> Depends on the kind of accident you have in mind, so I can't give real feedback here.
Typical scenario here is a dog appearing on your lane in a high-traffic, multi-lane road. You don't swerve, you don't even break too hard, because hitting the dog is much better outcome than hitting another car, or landing in a ditch, or getting rear ended.
Emergency breaking can typically stop a car within about 100 yards even from freeway speeds. But a distracted human might travel more than that before they even realize they should be trying to stop.
Humans are also pretty bad at adjusting to conditions, so if weather makes the road slick (wet or icy) or they have limited visibility, they don't often slow down enough to compensate.
Autonomous cars have none of these problems, so aren't likely to ever run in to a "can't stop in time" situation.
I can't actually think of any situation where, in the event of a likely collision, the best solution isn't to just try to stop. Even if, for some reason, you can't stop quickly enough to avoid the collision, any reduction in speed will result in a reduction in the severity of collision.
I don't think I that will be accepted by regulators/society until it can be demonstrated that AV's are undeniably safer-- not just as good as human drivers, much much better. Right now it's newsworthy whenever a Tesla auto pilot drives into a barrier. We'll need to get to a point where not only does that not happen, but also that it becomes newsworthy every time an AV saves a life-- swerving out if the way faster than any human reflexes could have reacted to avoid an accident, or a child chasing a ball into the street, etc.
Way to keep it classy, Waymo. Glad I don't work for you, I prefer to not be thrown under the bus by my employer.
That'd perfectly explain why the car would suddenly start trying stuff again once the roadside crew got close (and why it kept cancelling the roadside call).
Wow, that was cringey. I would have ducked out the passenger-side door half way through the video and walked.