Matrix has really interesting properties for censorship circumvention, but their approach has drawbacks: designing an append-only log with consensus building takes a lot of resources (though the implementations are getting better). It's not uncommon for medium server operators to report >10GB RAM usage and at least twice as much database storage... which is quite too much for most of us.

Also worth noting, there are also issues regarding the fast evolving protocol. Room protocol versions keep increasing and if some server doesn't have the very latest version it may be unable to federate with a more modern room, from what i understand. (please correct me if i got this wrong)

Finally, some pseudonymous persons are really unhappy that as a consequence of decentralized rooms, your public matrix address is leaked to any server federated with a room you joined. Under IRC, addresses are local to a server (not federated) so this concern does not apply, while on XMPP only the servers/services you explicitly connect to know of your JID.

In the meantime, IRC and XMPP provide useful services with 1-10% of those resources, despite most clients being less polished. For example, jabberfr.org uses only a few gigs of RAM for hundreds of clients and over a thousand of server-to-server connections. Not all features may be available on all clients/servers, but "graceful degradation" is definitely there so noone is left out.

To be clear, decentralized room is an amazing concept, and it's really cool the Matrix team has full-time employees to evolve things that quickly. It's just too bad historical standards were left out (decentralized rooms could have been a XMPP extension), and Matrix focuses (though this focus is changing) on high-end clients/servers.