Native Matrix VoIP with Element Call
element.io
element.io
Moreover, because it's built on Matrix, MSC3401 (https://github.com/matrix-org/matrix-doc/blob/matthew/group-...) means that we'll finally have decentralised cascading video/voice conferences once the SFU (selective forwarding unit) component is added into the mix. So, for instance, users on the same homeserver will get their video feeds relayed locally with minimal latency... and then users on another remote homeserver will also get mixed locally with minimal latency, trunking the two together. If the link dies or one homeserver dies, the conference will keep going - i.e. precisely the same semantics as normal Matrix.
There are a lot of performance/latency/sound quality comparisons online of Mumble vs TeamSpeak vs Discord and recently Jitsi vs MS Teams vs Zoom, etc. I feel like this is a problem-space that can be optimized to an arbitrarily deep extent. Just two examples that come to mind are SFU performance/efficiency and noise reduction, two things where e.g Jitsi notoriously lags behind Zoom.
* Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party.
* Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/
* Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too.
* Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important.
We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.
We can probably also be more bullish about supporting new audio codecs like Lyra.
EDIT: oh, the other obvious thing is echo cancellation - but we're currently at the mercy of the browser's WebRTC stack for that. However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.
1. I don't think there are many good open source SFUs to choose from. I know of 2, maybe 3 (including our new one). There may be many, but few have good rate control. But maybe I just don't know about them? I'd be happy to learn of more good ones.
2. Echo cancellation is certainly a hard problem, but it doesn't conflict with E2EE unless you do it on the server, which isn't necessary. So perhaps it may be a somewhat harder problem because you close off one possible approach (doing it server-side), but many (most?) echo cancellation solutions are done client-side.
3. You may not be completely dependent on WebRTC's echo cancellation any more because of the new MediaStreamTrackProcessor and MediaStreamTrackGenerator APIs. I don't know if it will work for echo cancellation, but it might.
2. From context I think you're talking about noise cancellation here? I assumed that some of the more exotic ML-based ones ran serverside, which obviously is incompatible with E2EE. It sounds like there are a bunch of options for running WASM-based intelligent noise cancellation clientside though, especially with MediaStreamTrackProcessor and friends. rnnoiseless as a pure Rust->WASM port of rnnoise looks fun, for instance: https://github.com/jneem/nnnoiseless
3. True, although given Google are highly motivated to make AEC work properly in WebRTC, I guess I'm hoping that they'll continue improving it, much as they have been. I certainly never want to have to write or integrate one ever again :D
Thank you! This is something I've wanted in a videoconferencing app since I discovered it was a thing, and I'm glad it's Element doing it.
Unfortunately the IP was all proprietary and owned by the company, and when WebRTC stole our lunch we shifted focus to messaging and building Matrix, and have slowly returned focus to VoIP - hence Element Call.
With Audio Worklets or insertable streams, however, it should be possible to apply it to the stream im real time, though we haven’t tried it yet.
(At least the smaller ones like interviews where the number of participants is <=8 until compatibility with SFU is done)
I'd love to replace my teamspeak server with this. Discord is tempting, but I'd rather host it myself.
You can self-host a Jitsi instance and use it in Matrix rooms as an integration. Depending on your teamspeak setup you can enable guest access to your server, enable registration for server user accounts (or manually register it for specific members), or only let people with existing accounts from other servers join in (their own, matrix.org ones, you name it). You might also need to tweak the ACLs a bit, but other than that you'll have all the core features Teamspeak and friends provide.
I think the biggest difference between the existing Matrix group call system and what's been announced here today is the new UI/UX, and the fact that the native group call API and reference implementation are nearing completion. Both are great news, but don't necessarily add new user-facing features to the Matrix ecosystem.
The jitsi stuff is bolted on, you join a link and your user name does not come with you.
No discoverablity. Does this channel already have a jitsi room attached, no way to know. I see a link to a jitsi room, is there anyone inside? No way to know.
Recommending this as a alternative to TeamSpeak, mumble, or discord today is a way to get people to try a horrible experience and then say Matrix sucks forever.
Only when this call stuff merges will Matrix start being viable for those communities.
With this we could finally have a proper Teamspeak/Mumble bridge for voice that gets properly represented on the Matrix side which is amazing :D
Also kinda funny that Teamspeak only recently started using Matrix for their global chat feature [2].
Maybe now after the gitter acquisition, Element should consider acquiring a certain voice-focused company ;)
**
[1] https://github.com/matrix-org/matrix-spec-proposals/blob/mat...
[2] https://community.teamspeak.com/t/teamspeak-5-beta-bug-repor...
* `m.intent` to describe the intended UX for handling the call. One of:
1. `m.ring` if the call is meant to cause the room participants devices to ring (e.g. 1:1 call or group call)
2. `m.prompt` is the call should be presented as a conference call which users in the room are prompted to connect to
3. `m.room` if the call should be presented as a voice/video channel in which the user is immediately immersed on selecting the room.
Element Call implements `m.room` effectively (in that you get slung directly into the call once you click on the link). Once we integrate this into Element itself, then the plan is to support all three different intent types: `m.ring` for "group calls", `m.prompt` for "conferences" and `m.room` for discord-style voice/video rooms.
I assume m.ring would not be needed if its a something like a Teamspeak room.
(N.B. we might not bother with the group-calling option in Element; haven't decided yet - but Matrix needs to support it for folks who want those semantics :)
So m.ring existing isn't a problem.
How it is exposed in the UI, is a completely different thing, although I don't think it would be impossible to make a button that creates a room with a m.room call aka voice/video room
The list of upcoming features is long and sometimes I don't understand how they can have so many things going on at once.
Features tend to take a few years to mature, but that is not a bad thing.
I will be setting up my own server shortly, something that I have been thinking about for a while and maybe try to develop something.
Congrats to the team.
Question:
The OpenID part, would it not be cool if Matrx would offer something like a federated SSO experience? So that on other webpages you could use your matrix login instead of Github/Google and co. Hosting that server yourself would be fantastic.
There are of course some issues with Attribute retrieval if user can host these servers themselves.
Yes, hopefully we'll be able to use the OIDC server bundled with your Matrix server to auth users in general if wanted (or alternatively you can of course point at an existing OIDC server).
https://aaronparecki.com/2018/07/07/7/oauth-for-the-open-web
The client would need better support for alternative inputs, of course, such as the RTMP suggestion found on the Github issues page, but it can and should work.
What I'm also curious about is how this fits in with the experiments on the peer-to-peer mesh system (running a homeserver on every client). Forwarding video across a web of clients would become bandwidth and CPU intensive real fast and the added latency would be non-negligible.
I'll be watching this play out with great interest, that's for sure.
Yes, we've built it with this in mind. There are three possible approaches:
* You can broadcast a headless client via HLS or RTMP, similar to how Jibri works for Jitsi, and how we broadcast FOSDEM (https://matrix.org/blog/2021/02/15/how-we-hosted-fosdem-2021...). Basically you run a headless Chrome against a virtual X server and pipe X's framebuffer into ffmpeg. It's a pretty blunt approach and obviously uses a lot of serverside resources, but ensures that the broadcasted stream is completely faithful given that it's literally a recording of what a client would be seeing.
* Alternatively, you could do the same approach, but pipe it into a WebRTC broadcasting platform rather than HLS/RTMP/RTSP/DASH. This could give much lower latency, but more bandwidth on the server given you're effectively setting up a 1:1 VoIP call with each viewer, and you'll be unlikely to be able to use CDNs (unless we end up in some crazy world where freeform multicast on the internet finally comes to pass thanks to CDNs fanning out your RTP packets for you :P)
* Finally, you could composite the streams together serverside using an MSC3401 compliant MCU and broadcast the result via one or other approach. This might not be quite as high fidelity as running a 'real' Element Call instance serverside, but could be way more efficient in terms of resources, and could prove perfectly adequate. The only catch is that we haven't bolted MSC3401 onto any MCUs yet. (Clearly we should get back in touch with our friends at FreeSWITCH / Signalwire :D)
> What I'm also curious about is how this fits in with the experiments on the peer-to-peer mesh system (running a homeserver on every client). Forwarding video across a web of clients would become bandwidth and CPU intensive real fast and the added latency would be non-negligible.
MSC3401 is pretty much orthogonal to P2P Matrix. The way it works is that it'll use whatever conferencing nodes ('foci') are advertised in the room to mix together the calls in a decentralised fashion. If there are none (as per today) then it just goes full mesh. If there is one, then everyone will converge on it. If there are two, then folks will pick the one with the lowest latency. But critically, it doesn't matter whether the clients are talking to a serverside homeserver or are running P2P Matrix. Finally, the foci could run either serverside or clientside (much as skype supernodes were effectively clientside foci), but we need to get serverside foci working first before we add clientside ones in for the P2P crew ;)
No it needs to be deterministic. "Just picking the lowest latency node" would result in sync failures i.e. not everybody hearing the same thing.
Yes, that is really important for classroom use cases.
The future SFU will similarly be split from the homeserver, with the initial implementation based on either Signal-Calling-Service, ionsfu or mediasoup (we're evaluating all three). Of course, the point of being standards based is that you'll be able to mix & match SFUs and MCUs from other vendors.
You can find out more about the architecture from my talk at Commcon (https://2021.commcon.xyz/talks/extending-matrix-s-e2ee-calls...) - or Robert's talk at FOSDEM: https://fosdem.org/2022/schedule/event/matrix_metaverse/
In three instances, I’ve had a long-running chats stop working completely and simply not load. All history/media inaccessible. Just a loading spinner. This is Element on iOS.
I wish it were more dependable, because things like this would excite me much more.
Seperately, we wre also experimenting with a full rewrite of the iOS app on top of matrix-rust-sdk called Element X, which whould provide a much better and faster codebase to try to avoid this sort of thing in future.
I wish I could update my parent comment. Although this interaction speaks for itself I think. Thanks for a great product!
How about a lightweight cross-platform Matrix client that doesn't rely on Electron?
Meanwhile, Nheko, NeoChat, Fractal and others provide very usable lightweight native cross-platform Matrix clients too. Element is also working on its own experiments (TBA) based on matrix-rust-sdk.
But then, I have decisively given up on Matrix ever being anything even remotely a solution for individual-individual kind of personal usage. But I do hope to see it being used at (my) work places soon.
There are multiple reasons for that. It costs money and I doubt the general populace will pay for that and since Matrix/element won’t harvest data and destroy privacy it’s a no go. They don’t have limitless resources like large corps.
Fragmented apps, using different severs (let alone running the server itself) in the too many clients isn’t “people scalable”. (It it ever even begins to try it it has to one one single blessed server and a blessed app - which at least the current such app Element is long time from)
So no, they can’t and won’t invest in making Element mainstream for personal usage and rightly so.
So what Matrix/Element is going to be?
Slack, Google Workspace Char etc: - YES!
WhatsApp, Signal, Telegram: NO!
But with Element now shipping chat bubbles as an configurable option, I'll stay with that. For me, the new chat bubbles size is perfect.
Aside of that, I'm generally really fond of FluffyChat, they are doing a great job!
Element Call sounds like something I should integrate into the conference experience, but logistics and audience size is a concern. We'll be entering 1k+ territory this year.
> In the near future, we will support using the app with any homeserver
I really hope this isn't an indicator how how Element the company will be focusing more on their paid offerings to the exclusion of supporting private homeservers. I've recently been experimenting with running a Matrix homeserver on a Raspberry PI in my basement--I'm excited about actual self-sovereignty in messaging but it isn't quite to the point where I can tell my family to rely on it instead.
However, we're in the process of moving Matrix over to OIDC for auth (as per https://matrix.org/blog/2021/12/22/the-mega-matrix-holiday-s...), at which point apps like Element Call should be able to easily hook into your existing account on your own homeserver using your existing server auth. In other words, the server will auth you, not the client. Combined with Sliding Sync (aka Sync v3) https://matrix.org/blog/2021/12/22/the-mega-matrix-holiday-s... this will then let you securely and efficiently use your existing Matrix account with apps like Element Call without it all getting bogged down with your existing chatrooms (or needlessly giving Element Call access to conversations it shouldn't care about).
Finally, this is beta: the auth/reg stuff is very much placeholder - for instance, it doesn't even expose password reset, given the upcoming shift to OIDC.
Complete sidenote, but just to expand on the above, the OIDC work is particularly exciting because (at least my understanding is) it opens the door for letting people use a single Matrix account for a lot of different "stuff" on top of Matrix.
There have been some interesting conversations and demos I've seen about using Matrix rooms to help with P2P sync for apps, and there are already bots on Matrix that use Matrix rooms for notifications so you don't need to send emails. But setting that stuff up is kind of challenging if you don't want to make a dedicated new user for each app or grant access to your account -- and (again, assuming my understanding is correct) I vaguely suspect that we might start to see a lot more 3rd-party apps built to integrate with Matrix once it's easier to point someone at a webapp and say, "just log in with your Matrix account, and it'll only have access to one room that's specific to this app."
So imagine potentially having a collaborative app that's hyper-focused on one task, like a collaborative D&D messaging platform or some crud. And in the backend, everyone using that app uses their regular Matrix account, and under the hood it's just sending messages to a shared room -- and then as a developer you don't need to worry about handling a bunch of encryption and user accounts and all of that stuff, you might not even need a backend at all.
On the basis that "the best code is no code", this could revolutionise app architecture. Imagine if instead of using blockchains, Web3 apps were backed by Matrix servers.
What does it mean that I can currently use it with my existing homeserver? The architectures for some of these federated systems can be difficult to understand, I feel like I didn't really understand what _Matrix_ was until I tried running my own homeserver.
The confusing part comes into play when the client implementation is a web app that you can use to log in to any server hosted anywhere. Because of how centralized services work, we are used to a paradigm where the website you visit hosts its own backend - you cannot choose. With these federated systems, on the other hand, you can go to website element.foo.com and log in at matrix.bar.com.
The blog post was saying they limited the option to log into your homeserver at matrix.bar.com (because of the reasons outlined above). This does not, however, stop you from hosting their (FOSS) app at call.bar.com and pointing it to your homeserver at matrix.bar.com
My family uses Signal for _a lot_, likely way more than we should, and i feel like we'd benefit from threads immensely.
Maybe i should toy with making my own Matrix client or something, heh.
edit: https://github.com/vector-im/element-web/issues/2349 looks like it might exist in some beta form. Depending on the client heh
Alternatives (Fluffychat, for example) are usually a while behind Element because of manpower. For example, native polls are in Element but not in Fluffychat yet. So, if you want to risk the transition, best to recommend the official clients first.
That said, I don't know how well threads will actually work in a family context. I like them as a concept, but outside Google Chat (where every message in a room is a response to or the start of a thread) I haven't seen them get picked up naturally.
Would love if there was a docker container to easily spin up an instance.
For instance, two users on the same LAN calling each other would end up bounced via the UK, which is a bit unfortunate if they are in Australia.
The solution is really to switch to using SFUs everywhere, which then solves both firewall traversal, scalability and privacy (assuming you’re happy for your SFU to know your IP - but if you’re happy for your TURN to know it, then it’s probably fine).
wondering if ip addys are exposed to the other users like with webrtc and if so, could we force conencts to be only through coturn to hide the ips?
Also wonder if there will be a warning for such, especially of encryption is turned on - some may think they are truly anonymous, and where ips are exposed of course that's not true.
Also wonder about moderation, hopefully this does not become a target for the sickening trolls soon - but moderation needs will be coming, so who gets the ip logs to consider blocking? homeserver runners?
I expect the future will need whitelists/blocklists subscription options for clients at some point.
However, this is utterly trivial to fix: matrix-js-sdk already exposes https://github.com/matrix-org/matrix-js-sdk/blob/96ba061732b... and we simply haven't exposed it as a setting in Element Call yet. I've filed a bug for it at https://github.com/vector-im/element-call/issues/251 - thanks for bringing it up!
In terms of moderation: this is no different to moderation in Matrix as a whole, where we're already busy working on shared greylists (MSC2313 and friends) - https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix... has more details at the end.
Also the post mentions: https://2021.commcon.xyz/talks/extending-matrix-s-e2ee-calls...
However, at this rate, things are looking pretty stable and i'd expect us to enable it in the next few weeks. It uses normal Matrix E2EE, which means a Double Ratchet between the pairs of devices participating in a given conversation, which is then used to secure the signalling which is used to set up the calls between the devices. The call media is transport-layer encrypted via DTLS and SRTP, so as long as the signalling with the DTLS fingerprints is secured by Matrix E2EE, the whole call fabric will be E2EE.
https://github.com/matrix-org/matrix-js-sdk/pull/2002 is the PR we actually need to merge in order to enable E2EE on it, ftr.
Hopefully by the time we've sorted out the SFU component, all the browsers will have RTCRtpScriptTransform implemented and so we'll get good cross-platform E2EE for larger calls (rather than it being limited to Chrome + Safari + Desktop)
I'll file a bug to make this configurable.
As usual, delusions of grandeur and horrifically over confidence in the abilities of a 45 million dollar funded gang of "developers" that have produced basically nothing in 8 years, when unfunded students have produced a discord clone in about 6 months
The main instance @ call.element.io also doesn't seem to work for me, but I chalked that up to (assumed) heavy server load.
we use jitsi common url to have weekly meetings. meet.jit.si/meeting2. everyone knows the url and time. we just join and talk. this is a nice concept
https://call.element.io/meetingwiththegang opens a meeting for a room named "meetingwiththegang".
Ideally, you'd want to use such a system with accounts so you can apply some kind of ACL, but if you want to use randomly generated room names for secret links then everything you need is there already.
Looks like you can't change the room name after starting the call, but you can start a new one at the new address if you want to change the name.
Turns out its much much easier to start from scratch with the right ideas than to transform an existing protocol in to something usable and convince everyone else and update all software to match.
"The market" has decided that Matrix works well and XMPP doesn't. If it made sense to fix XMPP then someone would have done it and we would all be using that.
Also 'the more federation and interoperability the better' is kind of contradictory to constantly reinventing extisting Internet Standards. I suspect VoIP over Matrix is not compatible with SIP or XMPP A/V calls and looking at current state of the matrix.org XMPP chat implementation they probably will never be.
I don't think tools that implement this exist in the XMPP ecosystem at all.
[0] https://github.com/matrix-org/matrix-spec-proposals/blob/mat...