I think the answer varies widely based on the underlying geology and what you're building.
Ancedote: My neighborhood in Lousiana was a land-filled swamp. Building a single story house there required driving dozens of pylons in to keep the house from sinking into the dirt. When we put a pool into the backyard, it had to be partially filled with water while the concrete was still hardening because otherwise the surrounding water table (something like two feet below the surface) would have popped it out of the ground.
This is probably less of a concern on, say, the Colorado Plateau.