I didn't read it as "we should trust people based on looks", but more as "we DO trust people based on looks" (I assume the "empirical evidence" refers to evidence of this being hardwired into us, not to evidence of it being useful).
The burden of proof is on those proposing a new system. I doubt it’s as easy as many rationalists assume.
The existence of the effect really means that it wasn't selected against.