It can also be a neurological or psychological issue. I had my hearing tested after years of being totally unable to pick out voices in a noisy environment. Turns out my hearing is actually above average. My ENT suggested that I might just be "bad at hearing", in the same way that one might have poor memory, or have poor visual-spatial awareness. It's apparently a skill that can be trained.