I've benchmarked it myself in my game. It adds a measurable amount of time to loads if you're decompressing all the audio up front to reduce latency, and during streaming it increases CPU usage. Obviously a modern machine can handle a few ogg/mp3 streams in the background during gameplay, but some games are playing dozens or hundreds of sounds at once.

"It's not like ENTIRE 35 GB of audio needed to be loaded all at once" misunderstands the nature of games fundamentally. Some games do potentially need the ability to access any of thousands of sound effects at a moment's notice with the lowest possible latency.