I think you're talking about different things. It's true, the Earth is substantially non-spherical. But things like mountains are tiny irregularities on a smooth (not-perfectly-spherical) surface.
If you stand at sea level at a pole you are much closer to the center of the earth than if you stand at sea level at the equator.