Why don't we see the emergence of local web search engines?
It seems feasible to handle large indices locally, even with periodic updates. Are there specific technical or commercial hurdles preventing this?
Also, local search is not scalable. Imagine if everyone, or even just a small fraction of end users, was trying to crawl the whole web for indexing.
But a couple of ideas that might actually work are:
1. A decentralised open source search engine, where the results are collated from a local index plus the indices of your connected peers, maybe plus a number of hops out to indirect connections. Would have to find a way to block users with tainted indices though, because that would quickly become part of the SEO market.
2. A local investigation engine without a central index to which you give a search query and which then trawls the net in a goal directed fashion to build up an index specifically with web sites that are interesting with regard to the search inquiry. You might have to leave it to run for a day or a couple, but you might be able to get good results with that, if done well.
But if there were a standardized index transfer format (that is, a standard index type and a wire encoding) then that would open up more options.
One could imagine an RSS-style feed where a site could post an index of its content. Or, if you think that's open to spamming, then multiple crawlers could publish indexes for types of content and users could subscribe. Hell, even something like UUCP or a Usenet style of "store and forward" could be used to propagate indexes across the network.
Still, the size of the index is a problem: I'm sure storage of the index would take many terabytes to make local search reasonable.