I remember this one bit from a video on self driving cars back a few years ago where someone basically said that self driving cars would not work well in cities because people will always want to mess with the cars just because they can.

Consider how in the early 2000s when people would mess around with AIM's smarterchild chat bot and gave it outlandish scenarios just to see how it would react. I have to imagine you might see something similar here with these cars but you're right that having a human onboard would probably curtail these interactions.