What kind of question is that? Of course I wouldn't be happy. Would you? What does me getting out of the car have to do with anything?
You could be car jacked while riding in a taxi/lyft/uber/etc. It’s unclear why you think this any different.
In this situation of yours it sounds like anyone can just walk up to a AV and open the door and get in.