I can attest to that. I run a Matrix homeserver for a small community, and it's blazingly fast all the time.
Everyone should migrate off matrix.org sooner rather than later. I fully expect seeing regional public Matrix homeservers popping up here and there eventually, just like regional public Jabber servers did back in the day.
Or, indeed like public e-mail providers did.