1. I spent 15 minutes last month helping my business partner sign onto ElementX on his second phone. He's signed on his PC and his primary phone, but the second phone was very difficult. Basically type username and password, but now you have to verify with emoji. How do you do that? Well, sometimes it pops up on your other clients that a new client needs verification. Sometimes the popup never shows, and there's no way to find it. But we found it eventually. Verify via emoji. Great! Let's do it. Oops, it timed out before he could finish. Apparently it has a super short timer that even I struggle with it. Eventually we gave up and used the recovery key, which worked, but you know many people are going to lose that key since it's not a password they can just memorize. Most other messengers, it's just username, password, and then click the email confirmation link. Boom, easy.
2. Audio/Video calling/conferencing. There are apparently 3(?) different technologies in Element alone, depending on which version of Element (desktop, web, mobile). The "old" one is jitsi. The new p2p one appears to be only available to clients that aren't Element? Or you can only create conference rooms in those other clients (commet?), but you can join them from any client. But then those new rooms are sometimes not join-able from Element clients. They just don't appear in the room list at all. Even clicking a link to the room says it's the wrong "room version", but it's a freshly-created room. So I delete the room and make another new one. But now it's a Jitsi room and it doesn't work in this other Element client (all first-party actual Element clients, and all broken in slightly different ways!). So... it's working now, but only after lots of trial and error and lots of cussing from my business partner. And I have to somehow justify that this is better than using Slack/Discord/whatever corporate thing.
3. The official client is fragmented. On mobile, there is Element (classic) and ElementX. You're _supposed_ to use ElementX, since it's the new one, but ElementX does not have all the features of Element (classic), so you might need the Element (classic) client for those things. But Element (classic) is not getting updated, so you're left in the cold as far as updates and the new features (meaning you might not even be able to cross-verify your clients, or join video calls, spaces, etc.)
4. The official matrix server (synapse) is pretty slow. I hosted a couple small communities (~100 people or so?), and synapse somehow kept getting overloaded and crashing. Maybe it needed more than 4GiB, but that seems kind of silly. There are decent 3rd-party clients (continuwuity is my current favorite), but there's no migration path from one server to another without wiping your entire database and starting fresh. So.... my main homeserver is kind of stuck on synapse.
5. Bridges. Yes, you can bridge anything to anything, but have you actually tried it? Interface is super clunky. You have to run a separate bridge service, make sure it swaps keys with the matrix server via a yaml configuration dance, then talk to the bridge's bot in a dedicated matrix channel with special commands that may or may not get your bridge working. I've used signal, discord, telegram, and irc bridges, and they're all a mess in one way or another. My most recent attempt was with the signal bridge. I made the mistake of enabling end-to-end encryption. One day, the bridge server restarted and lost all its encryption keys. Cool, now all the bridged rooms are dead. So naturally, I delete the bridge service and re-configure it from scratch. Now it doesn't work because its bridge bot username was already taken by the previous instance with the lost encryption keys. And I can't just delete that bridge bot user because _matrix does not allow deleting users ever!_. So now I have broken signal bridge rooms and broken signal bot users that I can't ever delete, even with the admin panel. So naturally I open up a postgresql client and delete them manually. Oops, now my matrix database is broken in ways I cannot even begin to comprehend. Because deleting users is not allowed. So now I just can't have a signal bridge. _maybe_ there's a way to make it work, but I've been programming and sys-admin'ing for over 20 years, and it's just beyond me. I'm going to have to (again!) wipe out my homeserver and start over, losing all my messages and contacts (again!). XMPP had very nice transport services, and Psi+ had a very nice UI for configuring and organizing them.
6. Multi-identity support is lacking in all the official and most community clients. If I have a work account and a personal account, I need to start two separate Element clients. Each one is a hefty electron app, and they all have the same systray icon, so I need to trial+error every time to open the right one to message someone. So far, there's only one client I tried that had proper multi-account, and I think it was commet. It was ok, but I hated that it was a flutter desktop app where every click felt super unresponsive. So I went back to Element. Most XMPP clients have multi-identity built-in. I'm not going to use my work identity for personal chat, just like I'm not going to use my hackernews account to message co-workers.
7. No message auto-deletion. Many people like to claim that Element is more secure than Signal. It is not. If you live in an oppressive country where things you say can easily result in prison or torture, you need more than E2E encryption. You need messages to be gone. You can't trust that your device or your friend's device won't get confiscated and forced to reveal your password. Even the United States has forced device unlocking at the borders. Signal (and Confide, and even Telegram) have automatically deleting messages, so even with the keys, there's nothing to see. Matrix stores messages forever. Aside from some complicated server-side setting, there's no way to automatically delete them. You have to right-click and manually delete EVERY message if you want future privacy. Nobody's going to do that. And I tried enabling the synapse settings for auto-deleting messages, but they didn't work. It seems that feature isn't really tested, or it's documented poorly.
8. The spam/CSAM problem when running a public community. The admin tools are a bit lacking, so without some considerable expertise, it's often not worth even having a community matrix server. So folks just make a discord instead, which has actual moderation tools builtin. Yes, I know there are external moderation tools you can install, but why should that be required?
9. The protocol is too complex to make full-featured 3rd-party clients. This isn't IRC where you can make a basic client in 3 lines of perl. You'll notice that the 3rd-party clients are all lacking in some features of Element, the official client. Even the official ElementX client lacks features of Element Classic. Most folks are stuck on the official electron-based Element client, which has a fairly clunky UX (try explaining to your grandma how to cross-verify clients).
This list is just off the top of my head. I could probably write a whole book on the problems with the matrix ecosystem. But again, matrix is the best we have. IRCv3 hasn't really caught on, and the need for an IRC bouncer to receive offline messages isn't commonly excepted in 2026. XMPP was/is ok, but it hasn't been commonly used since Google Hangouts pulled the bait+switch on us 10+. I keep thinking I should really just go back to XMPP, or start using delta.chat since "everyone" has an email address.
I often can't even get fairly technical folks (who write code by hand, etc) to use Element, even if I do all the server setup and just hand them the keys. It's too hard for most folk. It's too hard for me, and I've been maintaining my own matrix servers for 7+ years. I maintain 3 matrix servers right now. One synapse, and two continuwuity. NixOS makes it pretty easy to maintain servers, actually, as long as they're not open to the public.
And if anyone asks me "what should we use instead?", I'd probably say XMPP or IRC. And we know neither of those is perfect, either, and nothing open source and federated is as easy to onboard like Discord, Signal, or Slack.
But even though Element/Matrix sucks, it still works well enough for me to use it as my main communication. At least until I finish my own multi-protocol messenger. Which I started about 6 years ago and haven't had time to work on.
Disclosure: I wrote some of the code on Confide's backend back in 2021. I do not currently work at that company and am not promoting or un-promoting the app in any way.