It's clear that it's a psychological thing. If I were you, I would go watch 'Good Will Hunting' movie (and watch it again if I think I had already watched it) and substitute Hunting's negative 'personal experience' with 'mathematics related experience' and assume that I'm 'Will Hunting' (while watching the movie :-).
Comment
---
My personal experience is little different. I didn't have any negative experience/memory when I was learning mathematics (and science) at school; instead, I always felt comfortable with any STEM related course/subject. But I had a very miserable experience when I had to learn English as a second language[1] at school.
At that time, I started strongly believing in the idea that no matter what I do, I will never be able to learn English as ESL as it always felt like an alien subject (literally :-)) and there was nothing I can use to get a handle on it. No wonder, I felt nervous and hopeless in my English classes.
Somehow I was able to reach college (in my country it corresponds to the grade 11-12 of high school education in most western countries) with shear brute force[2]. In the 11th grade, for the very first time, I had an English language teacher who gave me some hope that one can learn and overcome all language related problems. Though he only taught English literature (which can be more intimidating when one is at beginners' level with lots of self-doubts and anxiety), his teaching style was really inspiring[3] and it changed my whole perspective about learning a new language. Later on, I realized that it wasn't my (or other students who couldn't afford tuition fees of private ESL teachers) faults that couldn't learn the language, it was the teachers' fault as they were not really qualified to be English language teachers, at the very first place.
Now I feel quite comfortable with English language[4] and afore-mentioned story is only a memory now, one from the distant past.
---
1- to be precise, it was my third language I had to learn at school in a non-English-speaking country; I lived in a deprived part of the country and had a financially humble background.
2- memorized anything English related to get into the next class and didn't let any English language related handicaps stop my progress.
3- at-least that's how I felt at that time. I distinctly remember, some of the students, I personally knew, were only finding insignificant flaws in his English language related skills.
4- I'm very far away from being an expert in English language but I'm very happy with my progress I've made till now.