Hacker News Reader

Kagi, DDG and Ecosia should get together to build a crawler that provides an index open to all search engines. They could price it at cost (maybe even as a non-profit). It would only provide raw data so each search engine can still add value with ranking etc.

View on HN

That would be great, but: - Index is only good when SEOs don't know what's inside. Buying index access assumes some level of answers given to the questions: "how do you build it?", "how do you rank what's inside?". With more than one company it's very likely to be public knowledge very soon. - Index is the secret sause of a search engine. Not sure if anyone would like to share that.

But overall I'd like to see it. Maybe some pre-index struture? Just a large db of a page metadata? So that search engines can add their ranking on top of it.

Exactly, not an index but a large database with raw html of pages. Not trivial with modern SPAs but still doable. Storage and compute cost is probably the biggest cost factor.

> Storage and compute cost is probably the biggest cost factor.

I wonder if "not an index but a large database with raw HTML of pages" would cost less if you could get content providers to maintain a distributed cached copy of their content at their own expense?

Reply on news.ycombinator.com