No matter what science people tell you, music is not math. It's much more complex than that. :)
After three semesters of majoring in music, I made things easier for myself as a student and switched to computer science, with a hefty dose of math on the side. (I also thought that might be better for employment prospects; I think I was right.)
I've continued to pursue music as well, but wow, those classes are intense.