I agree. Having a vehicle that requires me to take over in an emergency defeats the whole point of a driverless vehicle. I'd just rater drive myself then as I prefer to be a driver and not a passenger.
Basically there should be two modes. Full auto when I'm too tired to drive (or sleeping) and semi auto where it takes over in an emergency. And a third mode of full manual for off roading.