Zoom also use the Selective Forwarding Unit (SFU) architecture. Or a central server that kind of "mix" streams and send them to all participants, if you will.
Jitsi Videobridge does this by receiving simulcast streams from each participant. This server individually picks out streams and qualities that will fit in each recipient's downstream pipe based on measured bandwidth and configured priorities (e.g. you could choose to give more bandwidth to those who are actively speaking, or shut down all but the last N speakers video streams).
Yes, the available network bandwith on a Jitsi Videobridge could be a bottleneck. Each meeting need to fit on a single bridge instance. However, using common servers connected at 1 Gbps or 10 Gbps, it shouldn't be any problem to have meetings with substantially more than 2-4 people. Say a HD stream from most webcameras outputs 3 Mbps at full blast, with 50% overhead due to the simulcasting. That's over 2000 participants on a single server connected at 10 Gbps, all receiving all video streams in full quality.
For meetings with 2 participants the streams run peer to peer, so here you get the best possible latency and quality.
We've set up Jitsi on a VPS instance at a nearby provider, and have not seen any problems with meetings of 10-15 people. Stability and performance also outperforms all the other solutions I've tried. There's currently a bug in Firefox's simulcast implementation, so if you have any participant using this browser, this feature gets disabled for everyone. Even with this issue, I haven't noticed anything but excellent performance with this few number of participants.
The following article might also be interesting.
https://bloggeek.me/webrtc-vs-zoom-video-quality/