Lots of good answers, but there is another one that's important too: echo cancellation. Most telco equipment has very good echo cancellation hardware in it. It's not impossible to do a decent job in VOIP, but not all of the software does it well. This can be very important when you have high latencies or if you are doing conference calls.

The other main overlooked problem is automatic gain control in software. This one drives me crazy. Most people are too lazy/clueless to set the gain properly on their microphone, so VOIP client makers tend to use automatic gain control to adjust the input. However, it works very differently that what you are used to in a telephone. If someone laughs loudly, or coughs you end up losing the beginning of the next sentence. It can work extremely badly in conjunction with echo cancellation depending on the software -- to the point where the audio can be very difficult to understand.

It's probably my biggest gripe with Google Hangouts because there is no consistent way to shut of AGC on all platforms.

I've written VOIP clients before and it is possible to have incredibly good audio (much better than normal PSTN -- mainly because the codecs are better). But it's very tricky to get it to work on random hardware, with random network conditions and with users who don't want to configure their client optimally.