These inferences are developed over millenia of evolution. They are probably not wrong. Being able to judge trustworthiness, aggression, etc. based on a quick assesment of face is valuable for survival.
But if we then met a complete stranger or tribe whose face at rest, say purely through random variation of bone structure, happened to look more aggressive or happier or more sneaky than the resting faces of our friends, it would lead us to make biased and unjustified predictions.