>it's a rather negligible delay compared to moving it

That's an irrelevant comparison. If you booked a long flight, would you accept somebody pointlessly delaying it for an hour just because that's short compared to the duration? The correct amount of avoidable latency is zero, regardless of how much unavoidable latency there is.

It's also false; I have a good mouse, good mouse skills, and my taskbar sits against the edge of the screen giving it effectively infinite height (see Fitt's law). I can hit taskbar targets in around 300ms.