Yes, but pranksters frequently vandalize stop signs with messages like “stop driving” or “stop eating animals” and human drivers are smart enough to figure out what to do. I’ve never seen someone throw their car into park or toss their burger out the window upon seeing a confusing stop sign.
If you ever travelled by plane, surely you've seen people at the airport trash their perfectly edible food (snacks, etc.) upon seeing the confusing signs just before the safety checks, even though in many cases they would have been allowed to pass it through the gates.
We are just too much used to the existence of optical illusions for humans that we fail to consider them a possibility for attacks. Adversarial attacks on neural networks are simply that: optical illusions, but designed for neural networks instead of humans.