The problem is that they unnecessarily increase the crowdedness, and make it more complicated for people to get out on the lower floors.
Overall it's the same amount of stops regardless whether you take the right or the wrong one. But for some reason being in an elevator that stops without anyone getting in or out irks me.