So why can't you just say turn left or turn right etc... Just not being the one driving.
I've done exactly what you've described but also had the passenger telling me spontaneity saying go left right forward.
There is nothing inherent that says you can't do that with autonomous cars. But also it's likely that there will always be cars you can drive for fun just like you can still ride horses for fun.