ipsec for the control plane (SIP) might be OK, but you'll probably run into jitter and latency issues with the media (RTP/RTCP). Also RTP has provisions for packet loss, but end-to-end encryption won't like that. SIP already has an encrypted mode [1], but I don't know how well it's currently supported in the wild. The purpose-built secure media protocols are Secure RTP/Secure RTCP [2].
Getting just plain SIP and RTP through consumer, enterprise, and cell carrier nets can be troublesome. For example, SIP especially is NAT-antagonistic, and edge routers can try to be "helpful" and do half-baked proxying or deep packet mangling that just messes up your smarter VoIP server. Consumer ISPs have a tendency to block SIP altogether. Throwing in encryption complications may give you another layer of hard-to-diagnose headaches. (Of course, if you can get the secure layer through the net it might force intermediaries to keep their fingers out. Other tricks like using non-standard ports can help with that, too.)
Source: used to run the signalling/media dev team for a VoIP provder many moons ago, but we didn't bother with the secure modes/protocols.
[1] https://en.wikipedia.org/wiki/Session_Initiation_Protocol#En...
[2] https://en.wikipedia.org/wiki/Secure_Real-time_Transport_Pro...