Say FSD comes online or NOA is enabled for city streets. A driver without their attention on the road might not notice the mistake, and a dedicated attacker could navigate the car wherever they wanted and do malicious things (attack the car, harm the passengers, etc).
Sure, it's not a big problem now, but it could be a problem when the Tesla self-driving fleet comes online and it drives you to a back alleyway to get mugged and/or killed.