Not a medical practitioner here, but it might be worth looking into prosopagnosia, or face-blindness. It more generally includes trouble recognizing faces, but maybe also what you describe. It's sometimes hard to diagnose too, since there are many non-face signals that are useful in identifying people, e.g. gait, hair, voice, context, etc.