How does someone economically store tens if thousands of terabytes of data needed for the indexes of a large scale search engine?
And have the large server instances (lots of ram)?
And have the large server instances (lots of ram)?
i was spending 2500 a month just on indexers and had query traffic taken off my costs would have shot through the roof since you want query nodes to all be in memory cache and that was expensive back then. today i would have used some modern in memory distributed doc dbs instead of query masters with heavy block buffer caches. i learned a lot but lost my shirt :)
Wondering out loud if there's a crawl that will produce real-time updates via common news outlets..
The 20tb drives would be too slow for query though. Query really needs everything in memory to be fast enough.