While speed is a factor as well, you still need a lot more energy consumption to move a mass of rock&dust some distance away than to move a small rover. This immediately follows from the formula for work and the gravitational force.
The real problem is finding enough time and money to waste on such a fruitless endeavor. If that can't be managed it nullifies the entire problem of digging a hole on Mars since a Mars base is nothing but a waste of time and money. Wasting time and money is table stakes.
Before you get angry with me: I'm not saying this is the way it ought to be, but it's the way it is.