Thanks all for the responses - the whole “matrix doesn’t have voice channels” thing is much clearer now: it’s not the hard bit (voice/video conferencing!) but just the UX of how it’s hooked up. This we can fix :)
Being able to hang out in a room alone to signal that you are open to chat if someone wants to join makes for a completely different user journey from calling someone and interrupting your interlocutor.