It's possible you have a touch of prosopagnosia, also known as face blindness. It would make sense as you would recognise people you interact with more regularly by other things (e.g.: voice, or even body or posture as the grandparent comment mentioned), but unfamiliar people tend to me harder to identify.