>6'0, 30s guy that has had many old lady scream at me due to temper tantrums

Exactly. They don't hold you to a lower standard. If it was a teenager they might've cut some slack. Most people will cut an elderly person or a person who's very outnumbered by kids in the car a little slack. Etc, Etc.

So perhaps that person in the other car looked at you and decided for whatever reason you deserved to be held to a lower standard than they project upon everyone if no other information is available.