One hurdle is that the easy to mine materials have already been mined. We have been using progressively more sophisticated methods of ore finding, mining and purification.
Doing that the second time around would be harder. Though we can probably recover good quality materials from scraps.