Indeed, how were SpaceX and Scaled Composites able to do so much with such limited resources, 30 years later and having benefited immensely from NASA's pioneering work?

NASA could definitely be run better but let's compare apples to apples. I'm sure any given engineer at SpaceX has a computer on his desk superior to the entire world's computational capacity at the time the shuttle was designed.