Google should open source what actually matters: their search ranking algorithm
cdixon.org
cdixon.org
The zealous idea that open source and trade secrets cannot coexist is detrimental to the open source movement as a whole, in my opinion.
Increasing birth rates enables economies to grow without becoming stagnant and old, placing undue demand due to pensions. An exceptionally high birth rate creates food production problems, housing problems, and generally the 'sardine can' effect. Low birthrates create a stagnant civilization that will eventually reach a tipping point when taxes get so high to pay for pensions that the younger generations will protest.
Market freedom to the extreme leads to monopolies taking advantage, 'gentlemen's agreements' between companies to price fix and many other problems that rules and regulations try to control. Equally a controlled market develops problems in and of itself as it's unable to respond naturally to market fluctuations, and quickly becomes unruly and unstable, and as many of the countries who have held long-term controlled markets are testaments to, they develop massive black-market economies to deal with these.
The zealous burning of fossil fuels, aside from global warming concerns, can expel vast quantities of radioactive materials, sulfurous emissions and a long list of things we've accepted in western society as being more harmful than smoking to the wide populations health, but rules and regulations on air quality increasingly get more lax (at least they have been where I live). With zero fossil fuel production, however, our civilization would collapse (at least with an immediate stop to production), and I'm not just talking the halting of transportation vehicles, I'm talking the stop in production of the advanced polymers used in virtually all aspects of our lives. The majority of people wouldn't have clothes on their back . . . my wardrobe would certainly be a lot cheaper if my clothes were all genuine cotton and wool. Oh and farewell cheap elasticated sweatpants.
Every good idea taken to its extreme is a bad idea and will create monumental problems in its wake. Electric vehicles are amazing, however a pure-electric is a ridiculous prospect in winter when the capacity hits below 60% of room-temperature, suddenly your vehicle can't get you to work and back on a single charge. Zero integration of hybrids is, as everyone knows, damaging to the environment but also an extreme waste of entropy. A 1 tonne vehicle at 100km/h is a massive store of energy, and allowing that to be wasted is criminal, the amount of energy humanity has wasted on braking at highway speeds would likely run the entire human civilization for over a year.
http://googleblog.blogspot.com/2009/12/meaning-of-open.html (submitted here: http://news.ycombinator.com/item?id=1008908)
In it a Google open source zealot rants wildly about openess being ultimate, apart from search algorithms which should be closed. It also fails to mention all the times open has failed (communism being one of the more spectacular failures of open).
A cynic might surmise what they actually mean is content should be open so they can profit from their closed search algorithm.
Wait, what? Communism is a spectacular failure of centralized planning over the entire economy. It has nothing to do with open or closed in principle, and in every implementation it has been aggressively closed in terms of information flows and decision making.
Wow, and here I was thinking it was a form of government.
Communism in principle has absolutely nothing to do with the flow of information and/or decision making, and its implementations have only very little to do with its basic ideas.
A true communistic government has never even been tried, we've only seen a 'new boss same as the old boss' approach to communism with one batch of fat cats replacing another.
"Pure communism" in the Marxian sense refers to a classless, stateless and
oppression-free society where decisions on what to produce and what policies
to pursue are made democratically, allowing every member of society to
participate in the decision-making process in both the political and economic
spheres of life.
Like most ideologies, the term "communism" is gooshy enough to mean whatever you want it to mean - but at heart, communism is concerned with the matter of who owns and controls the means of production. It's a form of government insofar as the decision on who gets to control the means of production is ultimately a political decision.
Communists believe the people should own and control the means of production - but every real-world implementation has used government as a proxy for people.
I suppose it's possible to run a communist society via direct democracy on a very small, local scale; but good luck trying to scale that up to the national or even regional level. At some level you need to adopt some kind of more or less representative government. In the case of communism, the track record is definitely skewed toward the "less" end of the representation spectrum.
In other words, your claim:
>A true communistic government has never even been tried...
is a straightforward No True Scotsman fallacy. It's no different from "real" conservatives distancing themselves from the Bush administration because the implementation of their political system ended up violating most of its ostensible core values.
If you can't implement an ideology without discarding the very principles that define the ideology, it's a broken ideology.
To define American-style conservative and liberal philosophy as what the Republican and Democratic parties and their politicians do is ridiculously simplistic. The parties are just coalitions of interests and voter demographics, not philosophies - you might as well talk about what direction a plate of spaghetti points at.
And as for conservatives' "political system" - whose? Do you mean evangelicals and other theocrats, PNAC neocons, populist teabaggers, or Lew Rockwell/Ron Paul fans? We're not talking remotely the degree of unity imposed by the Politburo and Comintern upon the global communist movement when it comes to either US party. In a bipolar system, both parties must be big tents or be crushed by the bigger tent. Elected officials are not (and Bush was not) advancing a single "conservative" or "liberal" ideology; they're trying to please various power blocs who agree or disagree with them on different issues.
Bush's failures highlight the failure of particular policies and ideas, but they don't say much about the ideas of self-described "real conservatives" who thought Bush was a ridiculously free-spending, anti-federal, protectionist, power-consolidating, Big Government guy long before Iraq. (No more than Obama's decisions indict the ideas of supporters who disagree with him on those decisions, certainly.)
I don't think communism is evil so much as fundamentally naive, its proponents assume the mass of people think alike and that selfishness is a learned behavior imposed from above by wicked capitalists. It lasted as long as it did in the USSR because both military and scientific folk are into the idea of setting aside individual gain for a greater purpose, such as patriotism or the advancement of knowledge.
Communism is bad if you look at it from the perspective of a democracy with a functioning market. Then again, that perspective is a luxury if you have the misfortune to be stuck in a feudal society - most places where it took root were extremely dysfunctional monarchies prior to turning communist. Aristcracies aren't what we'd call ideal societies either; even here in the west, the replacement of aristocracy by parliamentary government was viewed by traditionalists as a form of godless mob rule.
And it's not that long ago that people were complaining that GPL represented a kind of license tyranny which would undermine proprietary innovation.
That said, I think the author's point is still wrong. As a community, we have to understand that these open systems are an enabler of progress, some of which will undoubtedly be proprietary. Part of openness is understanding that you're not locking people into using this stuff, and they're not locked in to how they use it.
I don't see it as hypocritical that a business can talk about being open and how critical it is to give back, while still keeping trade secrets to pay the bills.
I recently saw the CEO of the Economist group speak about journalism, and one thing he said (paraphrasing) was that the best defense of journalistic impartiality in the modern age is to turn a healthy (and diversified) profit. The subtext was that you can then risk offending any given party by doing good reporting without the appearance of favoritism, and without having a corporate (or nonprofit) master to please.
In the same sense I think that the best defense of free software is to allow companies to use it on their own terms, and give back when they can/will. The most useful open source projects are used (and likely abused) ad-nauseum, and trying to limit any company, even one that preaches hypocritically about open source is a recipe for them to tell the community to shove it and go hand money to <insert big vendor here>.
While it's fairly easy to preach the open-source gospel to fellow programmers and nerds that appreciate the value of this stuff, it's business people and non-tech-literates that oftentimes allow the checks to be signed, and the open source community needs to look to them like a group of enablers, not inflexible idealogues.
I don't necessarily think this is true. Sites that have high page rank, do so because other sites of high value link to them.
Even if Google open sourced their search engine, spam sites would still have a hard time getting high quality inbound links. The reputable and high ranking sites wouldn't have any increased incentive to link to low quality spam sites.
Keeping the algorithm secret, and in particular the tweaks that eliminate links which aren't expressing "I judge this to be a good site", keep Page Rank useful.
The quality of SEO analysts out there is so poor that I believe you could release the source code to Google's entire search engine and people would still get things wrong or believe in ways to get around what I assume is an extremely complex set of mathematically-oriented algorithms. It's a classic example of the blind following the blind, and those that typically have followers are those that are popular for performing SEO work on huge companies that would never actually need it (e.g. Coca Cola).
After reading the comments at cdixson.org by the author and reading openness article at Google, it seems like author is simply saying: "I agree they won't open it. But then they should stop talking sactimoniously about openness." which I agree with as well.
Here is an interesting perspective: "And millions of websites will have to continue blindly relying on a small group of anonymous engineers in charge of the secret algorithm that determines their fate." Have you ever been delisted by Google? Is there a transparent process to get back on the list? Perhaps transparency is what Google needs the most at the moment.
Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.
I don't know what could be done as verification for the advertiser, but the publisher should be able to do something, right?
Without proof that is nothing but handwaving, if they are as impartial as you suggest they are they've got nothing to lose by letting you know what their take is and how big a portion of their clicks are confirmed fraud.
I disagree with the statement that Google would be giving away everything by opening up page rank. Google's "secret sauce" is hardly page rank alone. Google is really secretive about this stuff, but they arguably have the world's larges corpus of multilingual text in the form of their html index, coupled with the world's largest fault tolerant compute cluster, the largest repository of scanned text,the largest repository of digital video, and probably the largest repository of digital audio in the world. I'm sure that they also have the world's largest repository of people's surfing and searching habits, along with the one of the largest sets of analytics data.
Even if page rank, map reduce, GFS, etc.. were open sourced I'm sure they'd still find a way to make billions off of the data that they have.
But, I think the larger point that the author makes, is that unlike many giant technology companies (Microsoft, HP, Apple, etc..), Google has made billions off of using open source software. And, while they are a very large supporter of open source, what total percentage of their profits go back to OSS? What percentage of their code gets re-released as open source? These are good questions to ask, and I'd love to hear an open discussion about it.
And does anyone gain from it? Not really. At this point, Google's supremacy is due just as much to their hardware know-how, to their market leader position, and to their excellent brand, as it is to their search algorithms. It won't help any competing search engines - just spammers.
The theory goes: if you let more people look at the algorithm, they can suggest better answers than Google alone, to thwart the spammers who are gaming the system either way.
But I suspect the real goal has more to do with the deep suspicion that Google can or does do a significant bit of hand-tweaking. And these people want to force the algorithm into the open to either end or curtail that.
Spam:email = 90% or worse
in search results it is closer to the opposite.
I'm sure once the open source community actually manages to do this and prove the 'many eyes' doctrine applies to more than just security and bug fixes that many doors will open.
Security by obscurity is a bad analogy here. The problem is that Google's algorithms are not mathematically perfect, in the sense of cryptography. Instead, they merely statistically tend to find good results. You can't find bugs just by looking at source code. You have to statistically analyze the effects of any proposed changes.
Compared to a lot of other companies (Apple, Microsoft, nearly every other tech giant), they are an open company. It's already great that they realize there's value in an Open-source browser, and open-source OS, and multiple open platforms.
This is a meme I see a lot, but never with any evidence. That the board has a fiduciary duty to the shareholders I can understand. That it would have a legal obligation to create value, well, I think that would be hard to define, and I have never seen any evidence for the existence of this obligation.
You can argue about the timeline (short versus long-term growth), but you can't say that there isn't a legal obligation to attempt to increase value.
I know they're not popular, or exciting like a stock value swing, but dividends still exist, right?
I actually think that more companies should give dividends.
I can see why companies like to have billion dollar war chests, but at some point you have to wonder when enough is enough. I mean, even Microsoft was forced to start paying dividends. Then again, their stock price has been pretty flat for the past few years, so that might be the only reason to hold it.
See Google's S1 (IPO) filing for reference: http://www.sec.gov/Archives/edgar/data/1288776/0001193125040...
Not to say it's interpreted correctly, but that is where the 'meme' stems from. If people quoted primary sources this might be more widely understood.
As some have incorrectly asserted, Google's latest post is not about "everything open", it is an internal memo to product managers that it occurred to them would be fit for public consumption.
The central thesis of the letter is that "open" is hard to define (and defined differently by different people) and yet crucially important. Because it is hard to define, many people misuse it (OpenDNS comes to mind), and no one can call them on it because it is a vague term.
Internally, Google employees argue about what open means, and the purpose of this letter was to clarify what open means to Google. They lay it out in extremely clear terms, and it is well structured.
This response essay by Dixon misses all that completely - he unilaterally defines "open" to mean "open source", and then asks for Google to give away their software, claiming that if they don't, they're hypocrites.
Going back to Google's definition of "open" (which is complex), one significant aspect they emphasize is the notion of "lock-in", i.e. locking in users, and locking out competitors. This is one area where Google puts their money where their mouth is; they let users leave. You can get at your email, export your Google Reader content, download all your Google Docs, archive your Picasa pictures, and export all your contacts. That means if a competitor comes along with something better, you can start using it. Try that with Facebook.
Dixon acting like that is worthless because it isn't the "open-source" form of "open" is patently absurd. If this is confusing, I highly recommend reading Simon Phipps essays on open source to understand what kind of software makes a good candidate for a business to open source (operating systems, browsers, compilers...the basics that we all use to compete) and what does not (grid/cloud computing algorithms, specialized software, and other types of software that are differentiators). Essentially, the business model is to pool our collective resources in areas in which we don't want to compete (web server software, operating systems) and to spend time innovating in areas we do wish to compete (search algorithms, voice recognition, image analysis, etc.) Over time, as those specialized areas become more mainstream, they become better candidates for open-sourcing.
In Google’s, case the correct relevancy ordering of the web is a huge point of contention. So they cannot simply open it up hoping to close out all the holes in the open. It is also hard to distinguish acceptable behavior (linking) from unacceptable behavior (selling links) except in the aggregate through heuristics that assume ignorance on the part of the attacker.
In this sense, Google’s system is a lot closer to an intrusion detection system than a lock or a cipher suite. You want to establish some ground rules but there’s no added security in declaring what you’re monitoring on your servers and how.
It's especially interesting if you consider the origins of PageRank in academia, freely published and peer-reviewed, and still not that well understood by laypeople; would the open-sourced magic of Google even be understandable by that many folk? Those that do understand it could make a fortune as consultants; at the moment, shady SEO snake-oil salesmen are so prevalent and 99% of their advice is outdated or plain wrong, something that definitively calls them out on it could be helpful.
In general people are willing to spend a certain amount on a solution. Therefore if you can things that make the complement to what you make money on cheaper, that increases how much you can charge for your piece of the solution. Open sourcing is a way to make those pieces cheaper.
Google does this. They give away tools and knowledge about how to make websites. They give away analytics and an A/B testing platform and educate people about how that makes it easier to produce good websites. This makes building good websites cheaper, and therefore increases how much people are willing to spend to advertise those websites.
This isn't a new strategy. In the late 90s when Oracle and IBM got into open source, this was their strategy. Oracle wanted to get people to stop paying for Sun hardware so Oracle could negotiate higher prices. (I know more than a few people who were surprised to find that licenses for Oracle on Linux are higher than they are on Sun.) IBM was looking to reduce the price for complements to IBM websphere, and was looking for opportunities to avoid having customers pay the Microsoft tax.
If you run a company and wish to strategically open source something, you should think in the same terms. What is the complement to what you are charging for, and how can you make that cheaper?
It's perfectly understandable. Lots of money is made in the existing paradigm and so many of you have incentive to believe it's the most rational system. Besides, hard to imagine Google's doing something wrong, right?
But the argument that "the only people this will help is spammers"? Really? Are the only people helped by open cryptographic protocols those who look to decipher traffic?
"They have every right to keep it secret and profit off of it." Let's allow this for a moment. We've still got a false dichotomy. Making it public will not deprive them of their profit. First, incredible marketing coup. Does anyone else remember how much of Google's initial strength came from its rabid fans? This would do a lot of re-energize that base.
Second, are there going to be new startups which can effectively compete with Google, even given its algorithm? How are they going to provide a better experience, or serve the volume of traffic, or index a larger amount of pages with the same algorithm with fewer resources?
Third, who says they have to authorize competitors to use it? This is I think one of the more interesting points. They could release the algorithm but license it only for non-commercial use for anyone not themselves. Ridiculous? Because Microsoft would violate such a deal anyhow? Because the algorithm could simply be reimplemented? Perhaps. But it would be another roadblock.
Fourth, why do we continue to believe that Google's strength comes from proprietary code? Why do we not recognize that its strength comes from mindshare, user experience, and quality of execution? None of which would be negatively affected by releasing the source.
Perhaps Google would face heightened competition as a result of releasing its algorithm. Perhaps this would outweigh the benefits of doing so. But there would be significant good which would result, including public feedback which would make it significantly harder for spammers to be successful.
But I'm just a foaming-at-the-mouth open source radical. And I'm sure history is on your collective side. If there's anything we've learnt from a hundred years or so of Computer Science (overestimate if you only count actual programming, underestimate if you allow Babbage as you should), it's surely that secrecy is what drives innovation, right?
I think it would be a good revenue strategy, if they charge $dollars to display top 10 positive features and top 10 negative features of any given query-document pair without any real weights/scores (maybe re-squashed with some logistic function). The $dollars amount is computed dynamically to offset the negative impact of the partial reveal of the ranking algo/model. This can be a tremendously useful tool for troubleshooting search results by site owners.
My impression is that PageRank, while initially extremely successful at filtering signal-from-noise, is very susceptible to manipulation. For example links farms which make unimportant pages seem important. As a result of such manipulation, Google started modifying their crawling, in order to reduce the amount of garbage going into PageRank (less garbage in, less garbage out). For example, they might look at the age and ownership in DNS records, and exclude domains that look fishy based on some heuristics. The spammers fought back with more tricks, and the resulting arms race has come to have an acronym: SEO. But the underlying idea of using an iterative approximation of the likelihood of encountering a page via a random walk of the web is preserved. (Some details here: http://www.miislita.com/information-retrieval-tutorial/matri...)
If that's right (and I could well be wrong... would like to be convinced...), then I don't see it as a "light-years away" kind of issue, but rather a series of hacks to stay one step ahead of the spammers. I don't see any particular value in open sourcing those.
Everybody could do it. This man is not asking for code, is asking for DATA only google has. It's like an atheist saying to a catholic what he must do because he believes. When this man gives something of value to the world,like google did(I remember the Altavista-Ad-flickering in your eyes days) then he could demand to be given by others.
Some balance in the force, for Yoda's sake.
Google, please keep it a secret, PLEASE!