I’ve tutored many students in math and it’s really common for people to feel the way you do. They’ve had bad experiences at some point or another, whether it’s with teachers or classmates.
There are a lot of different reasons why students feel the way you do and (unfortunately) give up on math. It’s a shame because I believe most of those reasons are specific to the high school setting and from what I’ve seen, students have a much better time if they come back to math later in life and study without that environment.
Interest also plays a role, of course. And not everyone will have the interest to go back and study again, later in life.