Sources: http://www.dspguide.com/ch3/1.htm
I think an additional layer of hiss would show up when it actually started playing.
Depending on how the original audio was mastered, it could also have it's own hiss in the sound data if the CD itself. You can tell if you load the raw CD data into a wave editor and see low amplitude noise where you'd expect silence.
For instance the jitter - the early CDs didn't have a super stable clocks that governed the conversion, so the slight changes of the pace when D/A conversion takes place would create slight distortion and it would also affect intersample peaks making it more apparent. Then you have the non-linearity of D/A converters themselves that were essentially acting like waveshapers.