This is the reason whey pumped hydroelectric energy storage (PHES) systems are so large. Gravity is quite a weak force.

https://en.wikipedia.org/wiki/Pumped_hydro