As with spam, the bad guys can be far more agile than the good guys. The point of having algorithms is that once you have a good algorithm, your search-quality staff doesn’t have to scale linearly with the number of pages on the Web.
(Disclaimer: I work in the search division of Nokia, i.e., I work for one of Google’s competitors.)
EDIT: Additionally, it is hard to game pagerank rapidly because the rest of the web needs to link to your site. So, even if you switch your knock-off wikipedia site to a new domain, it would take weeks/months to rise through the rankings. I'm pretty sure it would only take a few employees (at most) to stay ahead of these huge spam sites.
Joy's Law comes into play here: "No matter who you are, most of the smartest people work for someone else." 23,000 employees sounds huge, until you compare it with the million+ people who make their living as full-time E-bay sellers, and the however many million people who make their living off AdSense.
In the past there were people within Google who had the skill, knowledge, organizational connections and authority to quickly and gracefully reduce the problem of link spam without harming the bottom line too much.
I think we all loved their early work, and our love helped propel their little company to multibillions-per-year.
I don't think that kind of unique early experience and subsequent problem-solving effectiveness is snap-in interchangeable.
Some of those "rock stars" work at Facebook now. Google might not be capable of fixing the problem gracefully without them.
Not really. Some people own different websites, purchased through different accounts, hosted on different servers. They, then, link between those sites.
I would be happy if Google provided an "is spam" button that filtered the content only for me (i.e. without consequences to other users).
And as for the "is spam" button... a given email isn't automatically added to the spam training dataset the minute 1 guy hits the spam button. it goes automatically into YOUR spam folder but not automatically into the training dataset. That takes many "votes" from many users.
I see there are some browser plugins floating around: I'm not happy with those because I use multiple workstations, and because hacking around a product's deficiencies is not "voting with my wallet".
Same might be said for content farms, so I reckon your last sentence is on the money.
[1] http://www.reddit.com/r/IAmA/comments/azcni/i_made_62232296_...
I dont understand the legalities of that lawsuit though. A ranking is an opinion. Its not that its some fundamental physical constant or a formally well defined quantity. Can I be sued for my subjective judgment ?
OT: Will appreciate if you let me know why you down voted.
Consider that when discussing whether a company has a monopoly on a market, alternatives and cost of switching also comes up (besides market share); so it will be even harder than in the case of Microsoft.
Then even if Google is discovered to have a monopoly, shutting down a website ... can be argued that it was in the best interests of its users and faithful to the product's original mission: i.e. nobody can sue Microsoft for improving Windows / not bundling third-party software (they got sued for extending Windows with new functionality that destroyed competition in an existing market).
If you're willing to pay the tens of millions of dollars and take the PR hit that is a government anti-trust lawsuit, you may eventually have the privilege of making that argument before a federal judge, who probably isn't technology literate enough to comprehend it. Let's face facts here, this conversation goes over the heads of 80% of internet users. The judge is going to see "the federal government accuses them of restraining this site's trade by removing them from the search index, and Google admits to it" and then that's the ballgame.
Or maybe the government never gets around to suing Google. Or maybe they get a judge who knows this sort of stuff already. But that's still a scenario keeping Google's Legal Dept. up at night.
That would be a pretty stupid argument, even in the face of a non-technical jury, wouldn't it?
Of course, the negative image would hurt, but lets be honest, the suit against Microsoft accomplished basically nothing it couldn't handle, while waisting taxpayer's money. Are these antitrust suits getting started so easily?
Let's put it another way. I could conceivably build a valuable service by taking content from StackOverflow and Wikipedia (or wherever) and automatically linking the two to enable people to get some more context around some questions, maybe reformatting pages to enable side-by-side content or something similar. It wouldn't be a trivial service but it wouldn't be impossible in the least and it could plausibly add value. As such it wouldn't be unreasonable to preferentially direct users in some cases to that source rather than the original, as algorithmically optimised content - the preferential ranking would be a result of the value of the linking algorithm. Without this though, by what measure am I conceivably superior to the original source by having an out-of-date copy with fewer legitimate inbound links and more irrelevant content (adverts)?
Being a monopoly isn't illegal. Using your monopoly power in a way that might harm another company isn't illegal. What's illegal is doing that unreasonably and capriciously, as Microsoft were with Netscape in that trial, or DR and Lotus in previous trials.
No. That was incompetence.
1. What makes you claim blacklists are the "most-effective" form of filtering? My understanding is that other techniques like Bayesian filtering are.
Background: http://www.paulgraham.com/falsepositives.html
2. Saying that it only comes from a few sources is dangerous thinking. This type of thinking was used by politicians to get a ban on adult services from Craigslist ("get rid of it there and it goes away!").
There is a lot of evidence that there are not many botnets and each one has little diversity of control. This observation does not generalize to other kinds of spam such as 419s.