According charts in google, a jet engine
at 100 yards distance is 105dB louder than a whisper. 16 bits can do 96dB without dither.
More importantly, the problem isn't trying to hear 96dB of range all at once. The baseline is fixed in place, so when you have a quiet section all those high bits are 0 and you need the low bits to have enough detail by themselves. And a whisper is pretty far from the quietest thing you might have in a track.
If you can hear the dithering noise, then I'm pretty sure there are sounds you're hearing that would be wrong or missing without the dither.