One of the attributes of human perception is that it is TERRIBLE at maintaining persistent vigilance without engagement.
Even at a very low level, the nervous system is designed to habituate to constant stimuli; e.g., when you first encounter something that smells (good or bad) it can be overwhelming, but after a few minutes the same smell barely registers. More on point, spend some time looking forward at speed, or rotating (e.g., in a swivel chair), then stop quickly, and watch how your visual system creates the illusion of everything flowing in the opposite direction.
Now, scale that up to higher levels of cognition. The more the car gets right, the worse will be the human's attention. When a car here does almost everything right, people can and will literally read or fall asleep at the wheel. Until that one critical failure.
As a former licensed road racing driver and champion, I find the idea of anything between L2 and L4 to be terrifying. I can and have handled many very tricky and emergency situations at a wide range of speeds on everything from dry roads to wet ice (on and off the track) — when my attention was fully focused on the road, the situation, my grip levels, the balance of the car, etc.
The idea of being largely unfocused while the car does almost everything, then having an alert and having to regain, in fractions of a second, full orientation to everything I need to know then take action, is terrifying. 60 mph is 88 feet per second. Even a quick reaction where I've squandered only a half second figuring out what to do is the difference between avoiding or stopping before an obstacle, and blowing ~50' past it, or over it, at speed.
Attempts to say "it's just fine because a human is in the loop (and ultimately responsible)" are just bullsh*t and evading responsibility, even if human beta testing is fantastic for gathering massive amounts of data to analyze.
Among high risk and speed sports, it is almost axiomatic for us to draw distinctions between "smart crazy" vs "dumb crazy", and everyone knows the difference without a definition. The best definition I heard was that it's the difference between [using knowledge, technology, and skill to make a hazardous activity reliably safe] vs [getting away with something]. You can 'get away' with Russian Roulette 5 out of six times, and you'll probably get a great adrenaline rush, but you can't expect to do so for long.
Although this kind of "full self driving" has much better odds vs Russian Roulette, it is still unreliable, and the system of expecting the human to always be able to detect, orient, and respond in time to the car's errors is systematically unsafe. You will 'get away with it' a lot, and there will even be times when the system catches things the humans won't.
But to place the entire "legal and moral responsibility" on the human to 100% reliably operate a system that is specifically designed against human capabilities is wrong, unless you want to say that this is a [no human should every operate under these conditions], like drunk driving, and outlaw the system and the action of operating it.