I do that sometimes, unintentionally (regardless of the person in question having a beard or not). It's just that gender in language is most of the time redundant and it only introduces noise into conversation. For most topics, the gender of the persons involved is irrelevant. But many languages force you to choose one, which just creates confusion if this piece of information is crucial or if it's just this bloat that the language forced you to include in your sentence.
Oh, and genders are much more prevalent in my native language. English is actually better than most languages in this respect. But let's go a step further and just get rid of it. I've heard the Swedes chose to move to "person/thing" pronouns, as opposed to "male/female/thing".
In my native language, verbs change their forms based on subject's gender. And then when you have sentences like "if I were in your shoes, I would verb", my brain tells me to assume the gender of the other speaker when I say verb. I mean, it's only logical: if I were in your shoes, I would be of your gender. But the language works otherwise. It's just stupid.
As to people's reactions to my occasional misnamings, most people don't care at all, some are amused. Because, really, this part of language is ridiculous. And making an issue out of it is immature.