It seems clear that there has to be a single key, rather than separate keys for each pair of participants, since in a large meeting we need all video streams running through a server and everyone receiving the same streams, for manageable upload bandwidth. Also, many participants often cannot peer directly due to NAT etc.
But therefore... as long as you're trusting the server with key distribution/management in the first place... don't you necessarily have to simply trust that the server isn't peeking? That by necessity, since anyone else can join the call and get the decryption key from any of the peers... that the server can too, whether by MITM attack, a "fake participant" that joins for a millisecond but doesn't appear in UX, etc.?
That unless you actually have the capability of auditing the server and the code it's running, E2EE doesn't actually give you any concrete guarantees whatsoever? You've just got to trust? Which at the end of the day, is no different from trusting them not to peek at transport encryption?
Of course you can run your own Jitsi server. But if you've already got control over your servers then you might as well just be using transport encryption anyways, since you trust yourself -- right?
(Obviously if you come up with your own keys and send them to participants via a separate channel of communication then it's fine -- but obviously that's not something regular users are ever going to do.)
Would love to know if I'm misunderstanding something here -- if the newfound significance of E2EE in videoconferencing is just due to Zoom falsely advertising it, or if it's actually a realistic goal.