Comment by ChuckMcM | Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

Parent Full thread

That is awesome love that you've crossed a billion pages in the index. I worked at Blekko (VP Eng/Ops) and at its peak it had about 5 billion pages indexed, and that is pretty good for targeted search. It loses on the long tail though for generalized search.

Something we did at Blekko which was really interesting/helpful was to feed a search query to our index, Bing's, and Google's and then compare the results. That was a great way to compare notes on ranking strategies between the three engineering teams.

[1] "In 2015, Mojeek's index surpassed the billion pages, the only search engine in the UK to accomplish this and one of just a handful worldwide to maintain an index of this size (English language). Although, passing this milestone is only the first step and Mojeek is continuing to increase its index size and improve relevancy." -- https://www.mojeek.com/about/technology.html

Thanks for the info. Marc (who created Mojeek) mentions Blekko a lot as he considers yourselves to be the last ones so far to have a proper go at creating a search engine from scratch. Also, we're now up to 2.2 billion pages and it's only a lack of servers that stops us from increasing at a much faster pace.

For what its worth, this is the ultimate calculus for a search engine; The revenue you can produce is proportional to your query rate, your query rate is proportional to your recurring user base, and your recurring user base is proportional to the usefulness of your searches, which is proportional to the number of pages in your index which is proportional to the number of servers you have running which is proportional to the cost of running the service.

One of the things I would have done differently at Blekko is that I would have created our own in house ad serving system (network) way earlier in the life cycle (Blekko never did have an in house ad network). It gives you better control over advertising latency and the quality factor of the ads. You also get to keep all of the ad revenue rather than just some of it. And you can deal directly with advertisers rather than the ad-tech crowd which can be quite scummy at times.

I wish you the best of luck in your endeavor, if you can crack the secret of surviving in a market where Google pays people over $4B/year just to send their search traffic to them, then you will be one of my heroes.

Reply on news.ycombinator.com