SEO changed dramatically over the last 20 years. Thankfully, we are today exactly at a point that you described (it all started with Panda in 2011). As you'll see below, getting there was not just a technological challenge, but also one of fixing misaligned incentives.

Prior to 2011, Google enjoyed a mutually beneficial relationship with content farms which splattered their pages with AdSense ads (and Google ranked them highly). Can you imagine how it must have sounded for the Panda engineers to pitch to Sergey and Larry that they wanted to replace all those highly monetized websites with Wikipedia?

Matt Cutts commented that "with Panda, Google took a big enough revenue hit via some partners that Google actually needed to disclose Panda as a material impact on an earnings call. But I believe it was the right decision to launch Panda, both for the long-term trust of our users and for a better ecosystem for publishers."