I imagine the reason we get all the AI garbage is ads. kagi downranks sites that contain a lot of ads, while google, who obviously gets their income from ads, pushes them. that alone is bound to make a massive difference.

there's more, of course. but it's plenty for me to decide that i have no problem paying my $10 a month to not get 8/10 results that are ads.

Of course, I can't prove it, but the search results on Google look like they tried to win the spam wars with some variation of page rank that's skewed by "big brand power". Like they have some seed list of trusted brands, and graph proximity to that boosts your site over the pure spam. That, of course, penalizes genuinely good content that has no such proximity.

Kagi's various posts about their algorithm consistently talk about the value of "non-commercial content" and indexes to make sure they don't get lost. Like this post: https://help.kagi.com/kagi/search-details/search-quality.htm... Which sort of sounds like the same idea, but with more time and attention to curating a better list.

Google is selling ads - their search is designed to show you whatever will push more ads. They not only have no incentive to hide SEO nonsense if it has advertising, they are clearly breaking search to push more advertising in the results.

So I suppose the answer is that Kagi (and DDG, and Neeva-RIP) has (or have) no incentive to include spam, while Google is heavily incentivized to not only show it, but to make it appealing and promote it.

I think at some point in the past Google shifted it's stance of SEO from "if you do SEO you are cleary trying to game the system and will be punished" to "we are generally okay with SEO as that easens the burden of trying to guess content, and will only rarely punish it".

One instance that seemed like a turning point there was their reversal on the search penalty of genius.com after they were found gaming the system, something that I think was enforced much harder in the past.

So I think with embracing SEO spam to some degree (in combination with generally tailoring the behaviour of the search results more towards "consumption"), they must have lost parts of the resiliency. In contrast Kagi seems to be very alergic to SEO spam (AI generated or not), sometimes even too much.