Hacker News Reader

Comment by Semaphor | Hacker News Reader

This is bad for Kagi. They use both Bing and Google indexes, so every search is already costly. But Bing going up this much makes it crazy expensive.

View on HN

Kagi, DDG and Ecosia should get together to build a crawler that provides an index open to all search engines. They could price it at cost (maybe even as a non-profit). It would only provide raw data so each search engine can still add value with ranking etc.

That would be great, but: - Index is only good when SEOs don't know what's inside. Buying index access assumes some level of answers given to the questions: "how do you build it?", "how do you rank what's inside?". With more than one company it's very likely to be public knowledge very soon. - Index is the secret sause of a search engine. Not sure if anyone would like to share that.

But overall I'd like to see it. Maybe some pre-index struture? Just a large db of a page metadata? So that search engines can add their ranking on top of it.

Exactly, not an index but a large database with raw html of pages. Not trivial with modern SPAs but still doable. Storage and compute cost is probably the biggest cost factor.

> Storage and compute cost is probably the biggest cost factor.

I wonder if "not an index but a large database with raw HTML of pages" would cost less if you could get content providers to maintain a distributed cached copy of their content at their own expense?

I thought that Kagi had their own index too, and that Bing/Google are just sprinkles on top (even commented raving about that point less than 5min ago above). Is that not the case? It’s very possible that I just misunderstood their FAQ back when I read it?

They have their own crawler, and a side project with a non-commercial index they use. But the bulk of results is from Bing/Google. If those were just sprinkles, they’d have fewer issues, but their main cost is actually those two APIs.

Reply on news.ycombinator.com