The free matrix.org server is also overloaded. The paid server is much faster.
For example bridges break important features like end-to-end encryption.
Having 5 IM apps installed is much preferable to me than 1 worse app. Not sure the situation for XMPP but I have been told its highly fragmented with extensions that not all clients support. If I'm talking with someone, I want a high level of assurance that their client is pretty much the same as mine and that all features work and look roughly the same on both sides.
Internet standards do not innovate. People that adopt standards and built interesting things with them do.
For example, Google originally wrote the Jingle XMPP extension. Sadly, it seems like there are no real economic incentives for interoperabilty in the IM space (quite the opposite if you already have a lot of users on your platform), so we don't see a lot of investment going into the adoption of internet standards from big players anymore.