Maybe ask the server admins to make a voice channel category and then not only are text/voice separated/grouped but you can collapse the voice category when not using a voice channel. I really just don't see this problem, it's not even that I haven't done it but also I don't see other people doing it. To be fair though, like half the servers I'm active in organize their channels in this separated way.
Anyway, I think opt-in for some kind of friction makes sense.