I was under the impression that the dithering level is well below our threshold for hearing. Can someone confirm or deny that?
Sources: http://www.dspguide.com/ch3/1.htm
I think an additional layer of hiss would show up when it actually started playing.
Depending on how the original audio was mastered, it could also have it's own hiss in the sound data if the CD itself. You can tell if you load the raw CD data into a wave editor and see low amplitude noise where you'd expect silence.