For some roles, it's not petty. For example, for a sales role, you'd expect your salesman to be able to fit in at a client right from the first visit. If they can't do that at your company, that's a negative indicator for them.

Social signaling works both ways, and yes, in both ways, its petty.