Distributed karma: an idea for fixing recommendation systems
lkozma.net
lkozma.net
Our experience is that while this works very well against spam it does not stop the quality degradation that happens every time the community gets larger because the most active users tend to become friends and stop voting the news just for their quality.
If your goal is to create a system that reflects people's typical judgement, then this works, because people make all sorts of logical errors. If, however, you are aiming for a meritocracy, judging each post on its own worth without regard to who said it (except when identity is actually applicable), the correct approach is to have a swarm of AIs reading everything and assigning points based on content. [1]
[1] Implementing the correct approach is left as an exercise for the reader.
What you're suggesting is to remove the human altogether, and I'm not sure how AI would adjust when the average total quality reduces, which is usually the issue when a site scales.
Correct, which is why I mentioned the current situation works if you want a model that reflects typical human judgement, including bias and errors.
Removing humans is just a labor-saving feature. If you don't mind spending the time, then human editors can do a good job of it, just like with peer-reviewed journals. Make the score independent of the site's userbase size: e.g. from 1 to 10. No -47s or +581s like on reddit. Comments can also be kept in a useful range, instead of something blindingly obvious getting 80 points and something controversial getting 0 (+20,-20).
The key is always the editors themselves. Quality reflects the average of the editors, which is why you see a regression toward the general population's mean when everyone is imbued with editorial ability. I would be a very good editor for a few subjects and not good at all for most others; multiply that by a million and the overlap gets you the current situation with various social news sites. Pick editors randomly and you get Slashdot. Pick editors carefully and you get Science or Nature.
It seems a question of scope is in order here; what exactly is the purpose of a recommendation system?
Is it a system to forward to users that which they want to see, or is it a system that suggests various opinion of high quality to users? Often, I think we're trying to construct the latter by designing the former.
Secondly, Reddit's system works perfectly to forward to the user what they would like based on what's been submitted - the issue being that the average quality of submission has lowered over time. Even if the system gives you the best POS, you're still stuck with a POS.
The solution: Scaling is the problem, so stop/limit scaling.
We're not seeing a degradation of quality, we're seeing a better reflection of the average opinion - the larger the crowd, the lower the average. We're trying to enforce an expectation of quality that is held by a few on the many; this is impossible! The many don't hold the same regard or opinions as the few.
You can tweak things a little, perhaps come up with systems that use more cpu power than the space navigation does, but the end result will be the same: average opinion wins - exactly what you should expect.
Average opinion isn't what we want though, is it?
Haifeng Yu, Michael Kaminsky, Phillip B. Gibbons, and Abraham Flaxman, "SybilGuard: Defending Against Sybil Attacks via Social Networks." Proceedings of ACM SIGCOMM Conference , September 2006
http://www.cs.cmu.edu/~yhf/sybilguard-sigcomm06.pdf
In effect, this paper provides a way of "scaling" a trusted social network and minimizing the influence of sock-puppet accounts.
Reddit could/should have been using this kind of approach for ages (I don't know if they have or not).
Also, I'm just talking about delivering a good set of recommendations, anything else is gravy.
Here is a fun recent paper from google, about 'Scalable Online Collaborative Filtering' for their personalised news service [http://www2007.org/paper570.php], using MapReduce to run Expectation Maximization. via [http://www.datawrangling.com/google-paper-on-parallel-em-alg...]