There's nothing particularly difficult on the server side — a quality SFU should be capable to handle on the order of 400 video flows per core, and there are quite a few high-quality free software SFUs available (Janus, Jitsi, ion-sfu, livekit, Galene). To give some perspective: we're using Galene for lectures, and our single-CPU server uses around 40% CPU usage in a room with 120 students (who keep their cameras switched off during the lecture, of course, and only occasionally switch them on to ask questions).

As the grandparent mentioned, the problem is the client side. Since there is no standard videoconferencing protocol, every free software project needs to develop their own clients. And it's difficult for a free software project to have the manpower and expertise to develop quality clients for the web, Android and iOS, so in effect what we currently have are mostly half-baked web clients.

There is some hope, though. The IETF have been working on standard protocols for ingress (https://datatracker.ietf.org/wg/wish/), and if their protocols get deployed, you'll be able to use the same streaming software (think OBS) or IP camera with multiple distinct videoconferencing servers. An interoperable interactive videoconferencing protocol is nowhere near, but as more people understand videoconferencing technology, there is some hope that people will get together and start working on multi-protocol clients (remember Pidgin?).

Full disclosure: I'm the author of Galene (https://galene.org), and I've been actively participating in the Pion community (https://github.com/pion/webrtc) and collaborating with the authors of ion-sfu (https://github.com/pion/ion-sfu) and LiveKit (https://github.com/livekit).