What do you reckon the cost of doing this would be?
Just doing the napkin math for say a Mojeek sized index (couple of billion docs) doesn't seem to justify a particularly astronomical budget.
What do you reckon the cost of doing this would be?
Just doing the napkin math for say a Mojeek sized index (couple of billion docs) doesn't seem to justify a particularly astronomical budget.
That is indeed the key question. I tried to find out before commenting but the information I found is very vague. Internet Archive spends millions per year, but their index is updated far too slowly for a search engine. I have no idea what it costs to create a Google-size index.
Do you think the Mojeek index is good enough to compete with Google?
As a back-end for something like Kagi, it's sure getting there. Most of what sets Google apart is their exceptional level of user profiling. The actual indexing technology is likely on par with most of their competition.
Of course very little of that enters into API queries.
Assuming they'll need to index 820 billion pages (the number of pages preserved in the internet archive), at 100kb each, and assuming they use a database with 0.3x text data compression efficiency, they'll need at least 24600 TB to store those text data. Assuming $300 per 16TB disk, then they'll need to spend at least $7,380,000 for disk alone. This is a lot of money just for storage and we haven't included stuff like replication and backup, indexing metadata overhead, etc.
They changed that limit somewhere in the mid-2000s. Just as well, there's some CMS's out there where there's several hundred kilobytes of inline JS and CSS before any body text.
So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.
"The Google Search index contains hundreds of billions of web pages and is well over 100,000,000 gigabytes in size."
https://www.google.com/intl/en_uk/search/howsearchworks/how-...
Doesn't mean you have to be as big as Google to do something useful of course.
Google's index is likely very large because they don't have any real economic incentives to keeping it small.
Yes, absolutely, I didn't mean to imply otherwise. But first you have to figure out what you can discard beyond the HTML tags themselves to avoid indexing all the garbage that is on each and every page.
When I tried to do this I came to the conclusion that I needed to actually render the page to find out where on the page a particular piece of text was, what font size it had, if it was even visible, etc. And then there's JavaScript of course.
So what I'm saying is that storing a couple of kilobytes is probably not the most costly part of indexing a page.
The couple of kilobytes per document is the actual storage footprint. Sure you need to massage the data, but that almost entirely CPU bound. You also need a lot of RAM for keeping the hot parts of the index.
Are there open source projects devoted to this functionality? It’s becoming more and more a sticking point for working with LLMs. Grabbing the text without navigation and other crap but while maintaining formatting and links, etc
For my specific purposes it has always been good enough to apply some simple heuristics. But that wouldn't have been possible without access to post rendering information, which only a real browser (https://pptr.dev) can reliably produce.
Is that a good way to build an index?
An index is just hashmap of words and list of urls. So you have to parse the page, and add the urls and word frequencies to the list.
In terms of storage is cheaper, in terms of computing power is more expensive.
Hash tables almost guarantee worst case disk read patterns. You use something like a skip list or a b-tree, since that's makes much better use of the hardware, and on top of that allows you to do incredibly fast joins.
As for the index; this underpins mojeek.com and our API; which customers use for search and/or AI. Common Crawl is ~3.5 billion pages and underpins LLMs. Our index is ~7 billion. Who knows what (else) Google, and Bing, do with their index ;) ?
[0] https://blog.mojeek.com/2023/05/generative-ai-threatens-dive...
I'm sure you know your users well after so many years in the search engine business, but having read your article I must say I find your approach risky. You seem to be betting on search engines and answer engines continuing to be complementary rather than substitutes.
But we are not the ones making this decision. Users will be making the decision in light of the newly available AI capabilities, and they will be making it with complete disregard for the health of the web, as is their nature :)
The "funny" thing is that big publishers are as happy right now as I haven't seen them in the past 25 years, because it is so completely obvious that chat AIs will destroy the web unless big tech starts making big payments to big publishers. As you rightly say, small businesses and publishers will be collateral damage.
But how do you make sure you're not collateral damage as well?
An LLM to digest results of a classic search index is greater than the sum of its parts. An LLMs that is not permitted to brush up on the relevant literature before answering a question generally doesn't produce very good answers, is prone to hallucinations etc. A pure LLM design isn't even a serious contender in the answer engine space.
The big publishers certainly won't let you do that as they are selling their data to Google, Microsoft, Facebook and whoever else has the money to train a fully fledged LLM, which is certainly not everyone.
A search engine partnering with an answer engine may not send traffic, but the answer engine is a potential customer for the websites the search engines direct them to.
Only indirectly by charging search engines for access to content. It would be an entirely different business model that requires a complex set of agreements between publishers, search engines and LLM providers.
Granted it's not impossible and certainly worth considering if you have search engine expertise but no money to train an LLM.