Have you tried it with a different homeserver (either hosted or some other community server)? Matrix.org has a fucktonne of users (~5 million from memory), and given how little funding they have it's shocking that they can afford to run a homeserver at all. I've run my own homeserver for the past few years and I haven't ever had any issues with performance (even when talking to people on Matrix.org).