But that's extra time when the ship is sitting idle, while it could've already been on its way with new cargo, making more money.

Not so easy. Oftentimes there's no place to store the offloaded containers, and parking spaces are limited too. Probably a ship can also park "on the street" aka in open ocean, if weather agree, but I didn't see that.

Same thing with train carts that arrive to pick up the containers -- there's little place to park them to wait.