Comment by gzervas | Hacker News Reader

Comment by gzervas | Hacker News Reader

Hi everyone, co-author of the paper here. I wanted to clarify the 20% figure. We actually have no way of telling how many, or which reviews are fake. We do not directly observe review fraud, and we clearly spell this out in the paper. The 20% figure represents the percentage of filtered reviews on Yelp. These are reviews that Yelp finds suspicious enough to not publish. Some filtered reviews may be fake, and some might just be false positives. Similarly, Yelp's filter might miss some fake reviews, and end up publishing them. (See http://www.yelp.com/faq#filter_wrong). I hope this makes the distinction between fake and filtered clear.

Our main goal with this paper was to analyze the economic incentives behind review fraud, and for this we used filtered reviews as a proxy for fake reviews. bobf provides a good summary of our key findings so I won't repeat them. For those interested, you can read more here: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2293164

Wow, that is dramatically different than what the headline suggests. The "20% are fake" headline implies "20% of visible reviews", but instead I'm hearing you say, "20% of reviews are killfiled before visibility."

That's exactly my read.

And, it's not just the headline. Consider the following:

>A Yelp spokeswoman says its software helps filter many fakes before most users get to read them.

When combined with the headline, that really makes it sound like two discrete groups: the fake 20% that make it to Yelp, vs. the "many" that are filtered before making it to Yelp.

The difference is so vast that the headline now feels like link-bait.

yelp!

Filtered reviews are a terrible proxy for fake reviews!

Yelp is widely believed (among business owners) to wield review filtering/deletion as a club to "encourage" small businesses to buy advertising on yelp.com.

The economic incentives behind review fraud seem pretty obvious to me.

That's why we checked whether advertisers are getting preferential filtering treatment. (They aren't according to our data.) Read my previous comment, or better yet read the paper where we also explain possible pitfalls in the analysis.

You might want to rethink that then; I found one of my reviews filtered once, I have also been a "Yelp Elite" member so I am really unclear what criteria they use for the filtering. I also found some clearly fake reviews for the same place where my review was filtered. It was weird that they filtered my review as well because (I think) it was fairly balanced and mostly positive...

I don't know what goes into Yelp's filter either. I also agree that it can make mistakes. At the same time, I do think that the filter is more likely to catch a fake review than it is to catch a real review. This is the assumption our analysis relies on. (Not sure what I should be rethinking...)

Might want to re-think that exact assumption: http://justinvincent.com/page/1874/yelp-you-cost-me-2000-by-...

I understand that this is one anecdote, but I've seen enough similar cases of misapplied filtering to take it seriously.

While I read such anecdotes, I cannot rely much on them for research. Instead, what we did for our paper was to collect thousands of filtered reviews, and check their characteristics against reviews Yelp publishes. We find that filtered reviews are shorter, more likely to be from one-time reviewers, more likely to be extreme (1- or 5-stars), and so on. I realize this does not constitute absolute proof but at least it's evidence based on data. Check the paper for more stats.

Beyond that, and as a counter-anecdote, I have read hundreds (if not thousands!) of filtered reviews over the past year and in my experience they are more likely to be fake. Furthermore, one implication of challenging our assumption is that you are suggesting consumers would be better off reading, and basing their decisions on filtered reviews since -- by your assumption -- they are more likely to be genuine. As I said, I cannot offer proof, but given what I have seen I believe the filter to be more likely to catch a fake review than a real review.

From my own anecdotal experience, the filtering is crap. I just reviewed a local restaurant recently. It happens to be one of the best restaurants in town. It has only one review on yelp, a 1-star review, from someone who seemed to have unusual expectations. It also have 2 filtered 5-star reviews from people who loved it.

From what I can tell, all 3 reviews were kosher. Unfortunately because yelp incorrectly filtered the two 5-star reviews, the restaurant has an unfounded poor reputation on yelp. I left a 5-star review, and as far as I can see my review isn't filtered (at least, not when I view it). Perhaps that is because I'm a long-time yelp user.

I never got through all the pop ups to actually read the paper.

Reply on news.ycombinator.com