Telegram uses a custom protocol that makes professional cryptographers like Matthew Green tweet things like this: https://twitter.com/matthew_d_green/status/72642891296898252...
WhatsApp and Signal use the Signal protocol, which has a good reputation among working cryptographers (who I hope we'll hear from in this thread).