I would recommend looking at something like Mumble http://mumble.sourceforge.net/
Assuming you are concerned about the code you run, the location you run it in, etc. Does not provide encryption to my knowledge, but VPN can solve that.
If you are looking for something more robust that can interconnect with telecom stuff, I'd suggest http://www.freeswitch.org/ or http://www.asterisk.org/