The mistake is concluding it based on this type of scant information.
You treat it like any other unknown. You investigate deeper. If you work with the person for a while, you'll know his reasoning abilities far better than the information you gleaned from that hat.
>If someone walked in with a young earth creationist hat I would similarly question their ability to think critically (and have a hard time not just categorizing them as stupid).
And yet I've worked with a young earth creationist who was the best coder in his team (the rest of whom were not creationists). From my general experiences in life, his example is not much of an outlier.
Science Nobel laureates have believed in stupid things, as did many well known mathematicians of the past. Given that, why would you question his abilities?