If a subset of these inferences are shown to be wrong, wouldn't you want to make a conscious effort to stop making them? It would seem that becoming less biased in your evaluation of people would benefit both you and those you're evaluating.
And it's likely that a subset of all perceptions might always be found to not match reality. High availability consciousness would probably require artificial intelligence, and even then there's simply too much data for us to be conscious of everything at all times.
But if we then met a complete stranger or tribe whose face at rest, say purely through random variation of bone structure, happened to look more aggressive or happier or more sneaky than the resting faces of our friends, it would lead us to make biased and unjustified predictions.