The secure voice app is ZRTP (they are just using an existing open source ZRTP library). The ZRTP protocol is now an RFC: http://tools.ietf.org/html/rfc6189

The secure chat protocol they're using is something they developed and haven't yet published. It sounds as if it's similar to OTR, though. TextSecure (an encrypted SMS app I work on) uses a version of OTR adapted to the mobile environment, which is documented here:

https://github.com/WhisperSystems/TextSecure/wiki/Protocol

It sounds like a fairly standard shared key encryption. Basically, each user has a unique private key and all users share a common public key. When the data is encrypted, the combination of the private and public keys creates a number which can be determined only by someone also having valid public and private keys (or an immense amount of time).

Check out this video which attempts to explain this type of encryption: http://www.wimp.com/howencryption/.