By far, the most common reason I see it is how they were treated by math teachers. One student had a teacher who made students go up to the board and write their incorrect solutions on the board, and they had to stay there until they "figured it out". Many have been told they were stupid, or told they would just never be able to learn. That experience is way more common than many teachers and administrators want to admit.
There's also the pacing issue. Many students get lost in a math class at some point in their early learning career, and then there's no real way to get caught back up. This compounds with teachers who don't really understand math, who are just teaching algorithms and rules and don't know what to do with students who are lost.
I've had good success just by meeting each student where they're at, and letting each student progress from where they are. I always look for what students do understand, and build on that understanding. It typically takes students one or two classes being treated well to start really recovering from that earlier anxiety.