Unfiltered and unanalyzed, this is a bad metric because it essentially brings out the lowest-level mental evaluation of whether the person fits your tribe or not. It’s easily subtly biased against women, introverts, other ethnicities, etc. — not because you hate them but because your lizard-brain doesn’t automatically associate them with comfort.
Instead think about whether you’d want to hang out with the person, and then try to intellectually disregard that evaluation completely.