The car saw the obstacle and detected a collision, but there was no warning system to alert the driver about it.
Meanwhile, the driver was looking at a laptop and taking notes on the system logs (which they were told to do), not watching the road.
But the failure that killed Elaine Herzberg happened well before then, when the decision to disable the emergency brakes was made. It appears they disabled other safeties as well.
I don't disagree that playing a little loose with allowing new tech makes sense. We need to understand what we don't understand about them, accidents of course will happen, etc. But this looks a lot like foreseeable negligent homicide to me.
Like, let's say I had a grammar check. But the grammar check was bad. It flags every other sentence as needing a grammar change, anything other than a very simple "subject-verb-object" sentence structure looks wrong to it. For every correct grammar problem it identifies, it finds 20 incorrect grammar problems.
As a result, I turn off the UI for the grammar check.
Then I write a document that has a grammar problem.
It seems stupid to say that this is either a failure of my use of the software, or a UI failure of the software. The problem is that the fucking software doesn't work, specifically that it fails in a way that makes it too "noisy" to use.
And sure, if my job is to catch grammar errors, then it's also my fault. I'm supposed to be backing up the system. But we also know that I won't be perfect and will have failures of attention sometimes. But regardless of that, the grammar checker just didn't work.
And, in this analogy, bad grammar kills people.