Aren't they supposed to respond to gestures? What do they do if someone is directing traffic? And I guess the question if they do respond to traffic directing gestures: how do they avoid obvious pranks from drunk people?
Few are willing to don the reflective vest and ruin someone's day by lying to their face and directing them down a bad detour, but I have no problem believing that a lot of people would laugh at a stupid self-driving car wasting its time and energy following their hand signals.
There are valid reasons for <random citizen> to be directing traffic, like losing their load on the highway or helping traffic flow around a stopped vehicle.
Since there's no way to take control of these vehicles, I imagine they fall back to a service center with remote drivers when they come across the unexpected. Anything involving flashy lights should fall into this category.