I think the main issue would be the buffering existing tools are likely to assume. You don't need more than a few hundred ms of latency for a phone call to start feeling really weird.

If you went with the heavier-weight use of Asterisk, you could have softphones authenticated to Asterisk, but, no inbound calls from the softphones would be accepted. Only if 1 user sought to call another would Asterisk "dial" each softphone; which might be sufficient in terms of security (you would have to have a working SSH account to be called).