Basically, the sampling theorem says you can reconstruct the exact waveform with a certain number of samples. Adding a bunch more samples bulks up the file, but you didn't need them to restore the exact waveform. However, the unnecessary samples are in the file between you and the next sample you do need. At high enough levels of waste this creates an I/O bottleneck that hampers performance.

Another way to look at is that digital audio is not like digital imaging. There aren't pixels. Increasing the data rate does not continue to make the waveform more and more detailed to human auditory perception in the way that raising the pixel density does for human visual perception.

To describe it intuitively, forget your intuition that audio is like visual and start from "there is no metaphor between these two things."