Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/)?
Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/)?
Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm.
[1]https://googleblog.blogspot.com/2010/06/our-new-search-index...
[2]https://www.quora.com/How-large-is-the-Google-search-index-a...
[3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
A general purpose search engine would need that 99 petabytes of bad webpages to help machine learning algorithms classify the new and unknown web content as good or bad.
>if in some way this could run as a "personal" search engine. So I could fine tune it to avoid crawling and indexing certain sites
What you want sounds more like a "whitelist" of good sites to archive and a blacklist of sites to avoid. With the smaller storage requirements of the whitelist content, build a limited inverted index[1]. I agree that would be useful for a lot of personal uses but it's not really a homemade version of Google. Your method requires post-hoc reasoning and curation. Google's machine learning algorithm can make intelligent rankings on new content of new websites that don't exist yet.
I was just daydreaming, without first evaluating technical limits, that there could be a distributed search engine that runs on donated CPU/storage like those SETI and genome projects.
Thanks for the info.
The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results.
So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries.
E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon.
Compare IA with the "javascript" results from google.com[3].
It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors.
[1]https://en.wikipedia.org/wiki/Google_Panda
[2]https://archive.org/search.php?query=javascript
[3]https://www.google.com/search?q=javascript
[4]http://blog.archive.org/2015/12/26/the-internet-archive-tele...
Sure, expenses are a limiting factor along the way, but you use anecdotal evidence to make a point of that. Look at traffic instead, is google also single handedly stemming the whole infrastructure?
The question was about other search engines, non profit at that, and in doubt of that, about the possibility. Whether or not it would be expensive and whether or not the wikimedia foundation uses a lot of their money for meet ups and the like, there are possibly other limiting factors, that might ease the financial problem when overcome.
Is "technology advantage" another way of saying their PageRank algorithm of mathematically iterating on a linear algebra problem was superior to other approaches from Lycos/Excite/Yahoo? Well, yes.
But that algorithm has to run on thousands of CPUs which require lots of electricity and expensive programmers to program it. The algorithm has to be expressed on real-world things that cost lots of money. Google got $25 million of VC money in 1999. 18 months later, they figured out Adwords and that brought in enough money to self-fund expansion (buy more computers and harddrives). The IPO in 2004 brought another $1.9 billion to pay for datacenters. The superior algorithm must combine with expensive massive hardware scale to deliver quality results.
>The question was about other search engines, non profit at that
If you read jkaunisv1's comment more carefully, he wanted the homemade search quality to be "close" to Google's quality. You can't achieve that on a nonprofit budget of $12 million a year. (See my links for "javascript" to get an idea of what $12 million buys you.) To be fair to IA, their main mission is archiving and not cutting-edge search algorithm quality.