Not just the UX. It's dog slow. I've never had a good experience using it, no matter how much I want to like it.
Everyone should migrate off matrix.org sooner rather than later. I fully expect seeing regional public Matrix homeservers popping up here and there eventually, just like regional public Jabber servers did back in the day.
Or, indeed like public e-mail providers did.