Your stated argument against allowing human control of the vehicle is that it presents a safety issue, implying that you could be car jacked.

You could be car jacked while riding in a taxi/lyft/uber/etc. It’s unclear why you think this any different.