Instead imagine a system that is "good enough" that at it's most risk averse configuration it will drive itself in normal conditions, but occasionally it will seem to slow down for now good reason, or even stop, while it waits for a human to intervene because its not sure if its correctly reading the environment ahead.
It would need to be good enough to be able to work well enough of the time to be able to reliably come to a safe stop, but it would mean you'd be able to have one person monitor many vehicles.