All that means is longer lines and congestion of people waiting to board. So the bus is stopped for longer. This seems like a net nothing to me.
Additionally, many stops with a lot of people loading and unloading are hubs which would never be balanced away, and often are designated timing points where the bus will wait to get back on schedule, so loading/unloading time is often irrelevant because predictability is being prioritized over speed. Improving speed and consistency with techniques like removing unnecessary stops increases predictability and allows for tightening up timetables and minimizing average hold times.
[1] https://www.wmata.com/initiatives/plans/Better-Bus/frequentl...
The real killer for bus travel times is not getting up to speed, and the delay from finding a break in traffic when pulling out of a stop.
True I've seen that first hand.