I am going to venture and say this is not related to the ear apparatus, but in perception processing. I believe this type of deficit has been associated with ADHD, and other neurodevelopmental disorders/variations.
I agree with your assessment. I pretty much don't go to bars because of the suckiness. You can't have a worthwhile conversation, so it's just constant drinking in an oppressive environment.
That said, even in a quiet setting I have trouble understanding someone if I can't see their face/read their lips.