I agree on that extent. I find Telegram on this grey area which people recommend it as an alternative to Signal or WhatsApp (these people ask for group e2e) and at the same time many communities use Telegram channels and groups as alternatives/complementary to Discord. An app as Telegram is not going to please the majority and maybe this could be the case for Revolt.
Discord gets to 25,000 active users before they move to throwing resources at the problem for a hard max of 500,000 total users able to join (but not be active at once, that number is a little fluffy) so it doesn't seem e2ee is really slamming the breaks on scaling vs how far you could normally get.
Signals approach https://signal.org/blog/private-groups/
Whatsapps approach https://scontent.whatsapp.net/v/t39.8562-34/122249142_469857...
Finally if you allow tying and verifying to real world identity and don't require forward secrecy then the encryption side of the problem is no more difficult than PGP.