Hearing loss isnt so much an inability to hear "quiet" (despite how hearing tests are performed) as it is a los of dynamic range. By wearing earplugs you can lower the volume of all conversations and prevent "clipping", so to speak.
Try it! Next time you're having trouble hearing someone try to plug your ears with your fingers.