I honestly think this is a huge factor in violence against cyclists & pedestrians. Drivers think you're automatically beneath them in the societal pecking order if you're walking or biking, so they feel justified or entitled to victimize you.
People do similar shit to other people who they perceive to be poor, such as homeless, maids, delivery drivers, etc.
Although I haven't been subjected to any abuse about it, more than once I have had people make comments about when I'll be able to buy a car, as if finances are why I don't have one. The funny thing is that my income is in the top 5% of the area, and I have enough liquid cash right now to buy almost any car I want outright. I just don't want one.
People make weird assumptions.
People react with incredulity when I tell them I don’t have a car.
I live a 15 minute walk from the center of town and work and the UK has reasonable bus service in most towns so why would I spend thousands on a car and hundreds a month to park a vehicle I never use outside my house.
I cycle for fitness and pleasure though and most car drivers are lovely but we do have the odd dickhead as well.
Negatives? My wife doesn't feel comfortable riding the big bike with kids and there's only one bike with two seats. If the big bike goes into the shop, things get complicated but the same would go for a car-based commute.
It boils down to the fact that cyclists are unusual in most places in the US, and people are impatient and don't like dealing with things that are out of the ordinary.
But there's also just something dehumanizing about driving cars, which causes aggression. Road rage against other drivers is nearly as big of an issue as the issue of hatred towards cyclists.
Of course, even pedestrians get angry at slow walkers, but when they do so they aren't in command of multiple tons of metal that can move at high velocities.