In my experience, there's virtually no overlap between the two groups, and therefore no fragmentation. And for good reason: XMPP is a nightmare to implement, so there's a significant group of developers that just won't touch it, but that might be interested in working on Matrix.
And yes, part of the blame for that lies in the usage of XML. While XML can be useful to represent complex data or documents, it's unsuitable as an over-the-wire format because it doesn't have a directly mappable representation in most languages, due to the combination of attributes and child nodes.
This problem doesn't exist for JSON, because pretty much every language directly supports arrays, objects/maps and primitives. This makes a JSON-based protocol much more pleasant to work with, as there is less data-wrangling complexity involved.