If you find yourself constantly angry or stressed or nervous about other people you should probably remove yourself from the situation, relax and let it go. If you aren't prepared for confrontation or are worried about the situation escalating then use the moments before you let it go to come up with other ways to potentially solve the problem. Get crazy with it, like making a bunch of untraceable fliers, or making a long anonymous post to some digital community related to the location or activity. Their are lots of ways to influence how people act in the long term, in the short term direct confrontation almost always works.
Situations where you're unable to remove yourself from the area to avoid someone are the worst. This is why performances on the subway piss me off so much, or people ranting about something. I've been making a point to directly confront anyone doing anything that makes me uncomfortable on the subway and tell them why it makes me uncomfortable and that they need to stop. Just today some women came on the train and started yelling about eating a plant based diet, how it was good for the planet. I agree with her and I'm on and off vegan, but it's NOT fucking cool to start yelling on the subway unless you're getting violated by someone or you're in danger. I snapped her with my spectacles repeating, "ma'am, would you stop yelling on the train", ending with, "You're making all vegans look bad." She did finally stop