I think it would be the opposite. Someone who is taller than you would on average look more like their head is tilted up from your perspective, and vice versa for someone shorter than you.
(I'm going to observe what I'm doing myself the next time I talk to a shorter person :-))