There’s a simple reason there’s so much SEO spam: Adsense.
Maximizing ad profit on search traffic necessitates more paid ads, both on search results page and the web pages. They won’t say that directly of course but that’s what we’ve seen play out over last 20 years.
Vague metrics like “good user experience” get trumped by hard metrics like total $ / search.