Incidentally, I am a mathematician, or at least was a practicing one for some years and have published in the field...and I can say that from personal experience, math is exceptionally fragmented.
Yes, you point out a valid criticism but I don't really have the time to collect data on this.
Edit: I also wonder if the immediate denial of my statement is due to the emotional attachment that some people have to the purity of and beauty of mathematics. I also think mathematics is beautiful but that doesn't mean it doesn't have problems.