IT doesn't have to be that way. Ideally we'd stream the data (whether compressed or uncompressed) directly to the other end. There are standards for this but, well you know how standards are: so many to choose from.
In hindsight I think lossy compression of telephony was a mistake. It's 64 kbps for classic narrowband. GSM and other early digital cellular technologies could provide perhaps 5 - 10 kbps per handset and voice just had to be crammed into that. It made sense in the early 90s in that one application. It makes little sense in other applications, either then or now.
The long distance network of the late 70s into the early 00s was mostly uncompressed digital PCM, while the local loop was analog. The result was a basically distortion-free channel from about 200 to 3 kHz. Oh, and it was mostly synchronous too, so the delay was generally under 100 ms even cross-continent. You used to be able to immediately interrupt the person while talking just as in an real-life conversation. Some telephony systems running over packet-switching with buffering end up with such significant delays in practice that you have to take turns like it's a walkie-talkie.