New way to test self-driving cars could cut 99.9 percent of validation costs | Hacker News Reader

New way to test self-driving cars could cut 99.9 percent of validation costs

phys.org

> the new accelerated evaluation process breaks down difficult real-world driving situations into components that can be tested or simulated repeatedly, exposing automated vehicles to a condensed set of the most challenging driving situations

Chess engine programmers do something like this. Rather than playing thousands of full games to determine your engines strength, you can get a pretty good snapshot by running custom test-suites that contain a curated selection of known challenging positions [1]. One well known downside to this is that if you optimize your engine for test-suites, it invariably performs worse in overall general gameplay. Maybe we'll end up in a situation where cars learn to drive in an extremely over-cautious and risk-averse manner in order to pass all of the dangerous test scenarios.

[1] https://chessprogramming.wikispaces.com/Test-Positions#Test Suites

Perhaps I'm pulling teeth here, but at 70+ MPH, I don't think there's any such thing as overly-cautious until it backfires and becomes less safe in the process.

I think the "overly cautious" bit would be never thinking it's safe to come up to 70MPH in the first place.

Indeed, one thing I look forward to is SDCs returning residential streets to their original primary users: humans.

Self driving cars will actually take us away from a world where all car trips are humans accomplishing their goals, to a world where a decent proportion of car trips are machines serving their own purposes.

As an aside: please do lean further into this "people whose transportation needs are best served by cars aren't humans" rhetoric. It's going to be a lot easier to destroy at the polls.

I say this having just sold my car to move to a tiny studio in an urban high rise near work, because I recognize that my top-10% salary and lack of children are the only reason I can do that. I'm excited by the possibility of self-driving cars to bring the benefits of the automobile to more people, and most excited about those it can liberate from slow, crowded buses.

As derefr said I meant use of the roads for walking, playing, sitting, etc like they were before the "motor car" became common. The SDCs will have to be more careful.

Roads are now intended for cars, and new infractions (like jaywalking) were invented to give automotive use primacy over others. Cars should merely be one use of the roadways.

Not sure what you mean by "It's going to be a lot easier to destroy at the polls."

"Cars should merely be one use of the roadways" is equivalent to "people who can't afford decent urban homes should live in shitty urban homes." The primacy of the car is what lets us currently live in decent suburban homes.

Urban and suburban residents have competing interests here: urban residents (for whom cars are a nuisance) would like to safely frolic in the streets. Suburban residents (whose lives depend on fast travel by car) would like to continue affording decent homes.

This competition will appear in city-level politics in various ways. The anti-car side is handing the pro-car side some ammunition when talking about them as subhuman (i.e. framing drivers and people and mutually exclusive categories).

I think the parent was talking more about "robotic" traffic-law obesiance meaning that you'll stop hearing people roaring down your laneway at 3AM at 60MPH, just because there's no cars on the road and no possibility of getting caught.

Hi admin I examined your post about Facecord / Bushcord measuring Firewood. It was quite an impressive blog and it carries so much important and useful information, and I also liked the way you presented the blog i was amazing. Thanks for sharing! http://www.juegosdefriv.live/

Lets pass on the issue of memorizing the training data. The article also says the test is designed

   "to consider human drivers the major threat"

I'm thinking that degenerate driverless-algorithm behaviors will be even more problematic. Imaging an imminent head-on collision between two driverless cars. They could play 'dance with me', each dodging to the right or left only to see the other car do the same. In a few milliseconds they could lose the opportunity to avoid the accident.

With a human driver and their limited response time, the other car is actually more predictable and interactions easier to manage.

You can make a rule, just like ships already are doing:

> When two power vessels are approaching head on,both vessels should alter course to starboard to pass port-side to port-side.

> When two power-driven vessels are in crossing situation on a collision course, give way to the vessel to starboard (right).The give way vessel must take early and obvious action to avoid a collision by either stopping or altering course to starboard.

https://maritime.college/Boating-Rules.php

---

Plus, driverless cars can potentially use more efficient short-range radio and figure out themselves.

This was a problem with auto-dimming headlights for cars before computers existed. Patent applications would flood in from people who hadn't considered the interaction, year after year.

Furthermore, there's no hint in this article that in these simulations, the autonomous vehicle is being considered as potentially a threat to humans. I hope that is just a journalistic oversight...

it would take nearly a decade of round-the-clock testing to reach just 2 million miles in typical urban conditions.

...if you only use one car.

As stated in the article, they need to reach 11 billion miles to get the confidence level they are looking for. So extrapolating this, it would take 5500 cars a decade of round the clock testing to actually meet the guarantees they need (90% safer than human driven at a 80% confidence level). You still could do it in parallel; it'd just require a lot of test cars.

One part that is unclear is that the 11 billion mile requirement doesn't specify urban conditions and the 2 million mile per decade estimate does. So there might be some errors there.

Based on estimates that I have heard, the cost of a single car can be upwards of $300K. This limits putting large fleets of cars on the road only to the well funded (Google/Waymo, GM, Ford, Uber, etc) for only testing purposes. Smaller upstarts would need those cars to start taking rides and generating revenue a lot sooner. This seems like a worthy innovation to bring down costs and encourage more competition in the space.

And then, car makers will try to detect these tests and make their cars more cautios if in such a situation. Its probably going to be great for training but testing should be run in the real world, could finally also reintroduce realistic benchmarks for mpg and other running costs.

There will always be unanticipated circumstances that will foil the sensing systems. My go-to example is this situation where road re-paving artifacts and shadows combine to create a false lane marker. This vehicle's video-based system determined that the bright area between the shadow and lane pavement was the left lane marker and sounded a lane departure alert. Had the autonomous lane following feature been activate, it would have reacted and attempted to center the car on the false boundary:

http://imgur.com/a/KnsXU

Hmm.. Randomly generated real-world traffic based from traffic sensors and cameras. The key here is, if they use the same data and feed every car the same, then it can be easily gamed with overfit techniques.

The better question is, what happens if you give the AI car a no-win situation? How does it handle it? The Kobayashi Maru can be looked at as a way to detect outliers and failure modes... even if the game is rigged.

Unfortunately, the linked white paper is very thin on methods and analysis. It, itself, is not peer-reviewed, and it points to only one peer-reviewed publication to justify the approach.

seems like a good dataset for training a deep learning system.

This needs to be an out-of-sample test.

Reply on news.ycombinator.com