Yes, but the failure mode is not so easy to see prior to use. It's due to how humans react to the new feature itself, not because it's a bad feature, or a misunderstanding about how users 'normally' react. User's may incorrectly anticipate a behavior (such as in this case), or they may expect the feature to fail so unnecessarily take control (cause of some airline disasters).
This is a general problem of computer human interaction. I'd say it falls under the discipline of UX design. Knowing that breaking is an automatic reaction should be a clue that it will make it a poor mechanism to understand user intent. You then know that you should be looking at other mechanisms to more accurately capture user intent.
- Perhaps the auto-breaking feature should have better defaults such err towards enabling rather than err towards disabling.
- Perhaps the control for toggling auto-breaking should not also be the breaking mechanism itself.
- Perhaps context is now relevant, and user input should be interpreted differently when auto-breaking is on and there is an immanent collision.
I think it's important to pay special attention when making any interface change to a well established user behavior, and acknowledge that new behavior leads to unstable and unreliable input. In these cases put extra effort into finding the path of least cognitive burden when the accuracy of user input is the most critical (i.e. immanent collision) even at the expense of adding cognitive burden elsewhere.
Cognitive burden is a critical consideration in all system design from airplanes to cell phones, but tragically is often overlooked.
It is not easy but it should be done. Keeping a car steadily between lanes is not easy either, or take self driving cars that is hard. Besides, research into some of this has already been done with aircraft and pilots.
> It's due to how humans react to the new feature itself, not because it's a bad feature, or a misunderstanding about how users 'normally' react.
It is a bad feature. Feature doesn't exist in a void, it exists only because humans are using it. "Human presses the brakes, what to do next?" is a feature with a human in the loop.
> Cognitive burden is a critical consideration in all system design from airplanes to cell phones, but tragically is often overlooked.
Exactly. Moreover "mode confusion" (human not understanding what mode the aircraft autopilots or other systems are in, and then reacting assuming a different mode) is a common cause of crashes.
There were tools and studies specifically geared towards mode confusion:
http://shemesh.larc.nasa.gov/fm/fm-now-mode-confusion.html
Asiana 214 is a recent example of pilot assuming the autothrottle was on, but it wasn't, so plane crashed. It is a bit like here, the user assumed Tesla would slow down some but then stay in the automatic mode.
I'm trying to think of a scenario outside of a mechanic's garage that emergency braking would ever be disabled. This should really be mandatory on all cars.
In the lab, opaque textured objects in sufficient lighting are pretty easy to localize by a variety of methods. Reflective objects are quite a lot harder, but there are methods that work. Reflective objects through distortion like rain/fog... harder, but often doable. In low light? Depends.
The weird thing with environment modelling in computer vision is that there are a lot of overlapping methods, but sensor fusion of these methods is rarely attempted. Most robots end up with a pair of laser rangefinders or a pair of cameras or something equally simple to program. Creating a high-reliability automated vehicle that can bypass all the edge cases where these things fail is another thing entirely. You want overlapping, inter-reinforcing models of the world, with algorithms that not only identify items in a well-calibrated system, but calibrate the system in real time based on items identified.
My lab was really excited about replacing stereoscopic computer vision with monocular computer vision for 3D modelling, but the opposite direction is where you want to go if you care about as many 9's of reliability as possible and hardware is cheap. You want a few dozen cameras with fixed orientation, you want parallax laser rangefinders in a variety of bands, you want time of travel LIDAR units, you want a {GPS + 3DoF gyroscope + 3DoF magnetometer + 3 DoF accelerometer + barometer + thermometer}†, you want ultrasonic and radar rangefinders; You want everything you can get. You want enough lenses that if one gets obscured by water, hey, there are others that will pick up the slack, and do it without bothering the driver.
You want to shrink the 'no coverage of object in environment' surface as small as possible.
Nobody seems to respect this on the academic software side. Novelty, sure, but people don't seem to earn grants publishing extra 9's using a mashup of a bunch of pricy sensors. 'Less is more' is not how you get the data needed to make things reliable, it's how you get best-case detection rather than worst-case awareness.
†This combination is now found in quadrotor IMUs. Each individual sensor is inaccurate enough that it would be useless for automation, but combine them to check each other's deficits and you get a remarkably elegant system.
I'm pretty leery of these systems at the moment due to people using them like this. That's just not what they are ready for at the moment and people's unrealistic expectations are leading them down dangerous paths.
But otherwise, even when in manual driving mode, the emergency brake feature should be active and watching for impending collisions.
Also, it was a UI failure not notifying the driver clearly that auto-pilot has disengaged. Even the primitive cruise control in my car has a big green light, that turns off when you tap the break. The more autonomous a car is, the more important it is to clearly notify the driver of control handoff.
BMW's anti-collision system is on even during manual driving.[1] BMW will brake even if the driver is pressing the accelerator. So will Volvo. Top Gear compares the various systems as of 2015.[2] Volvo did best; Mercedes didn't stop soon enough and hit the dummy car. VW's city car only does this up to 18MPH.
Maybe Tesla has a false-alarm problem that BMW and Volvo have overcome.
More to the point, this behavior needs to be standardized, or people shifting from one car to another will do the wrong thing. The National Transportation Safety Board is working on this.[3] In a few years, anti-collision may be as standard as anti-skid is now.
[1] http://www.theverge.com/2015/1/5/7494989/bmw-i3-self-parking... [2] https://www.youtube.com/watch?v=PzHM6PVTjXo [3] http://www.ntsb.gov/safety/safety-recs/recletters/H-15-008-0...
Even though my car is a 6sp manual, the automatic distance keeping is making the commute much more relaxing. I can't recommend it enough.
Funnily enough the radar system is separate to the AEB, which uses cameras. AEB stands for automatic / autonomous emergency braking.
As I'm someone who likes to know how stuff works, especially in cars for familiarity should a real event occur, I did set out to experience how AEB works. I rigged some large but light cardboard boxes, reversed down the road a bit, accelerated quickly to around 40 km/h, kept the throttle pedal steady as though if to drive through them.
All times, although it felt like surely it wouldn't, the car intervened at the very last second. The first time it drove through them by a metre. The next two times the car stopped before impact. System uses 100% brake force engaging the ABS and also putting on the hazard lights for good measure.
I came out highly impressed. It ought to pick up cars better, but I'm not about to actively try that.
An appropriate way to signal it would be distinct from any emergency controls and also show a highly visible signal that you've turned the system off and are in a different "mode"; even a simple separate dashboard switch with a light would be better for it than the brake pedal.
So there's an accident, and I swerve to avoid. I think I'm going to make it but the car isn't certain so it brakes, and it brakes really fucking hard. At this point who is responsible for what happens? Breaking like that fundamentally changes the behaviour of the car. If I had just braked, maybe I stop in time maybe I don't - I chose another option and it is completely valid.
The very idea of that kind of behaviour terrifies me. I'll take a 100% automated car with pleasure, but I have no interest at all in this hybrid control concept. I would turn that shit off on delivery.
I'm having problems understanding your scenario. How exactly does the car breaking prevent you from swerving? The brakes should only stop your car mid-swerve, and you wouldn't collide. The only scenario when speed is an advantage when swerving is when you're trying to needle into an opening in the next lane and you need to match the speed of a 3rd car
It all depends so much on various variables so I find it hard to categorically say "it should always be on."
A deer crossing the road in icy conditions. Its better to hit the deer than lose control.
a large tumble weed blowing across the road with someone tail gating you.