Hacker News Reader

Comment by mcpherrinm | Hacker News Reader

I find the 5 star system on Amazon useful. I don't always look at the average rating, but rather the distribution graph of ratings.

For example, one common shape looks like this:

   5 stars: |||||||||||
   4 stars: ||||||||
   3 stars: ||||
   2 stars: ||
   1 star:  |||||||

There's way more 1 star than 2 or 3. That usually indicates the product shows up DOA for a lot of people, or is otherwise probabilistically totally unsuitable for some group of people.

If the number of star ratings follows a more normal curve like the following, then I'm more confident in the quality of the product, even if both have an average 4 star rating:

   5 stars: |||||||
   4 stars: ||||||||
   3 stars: ||||
   2 stars: ||
   1 star:  |

View on HN

This would make a fascinating study of reliability - can you determine statistically if a product will be reliable based on rating distribution.

You could study Amazon, App Store ratings, etc. Cool insight!

It's been done, and I've seen it as a worked example in introduction to Bayesian stats pages, maybe some of which were posted on HN.

I wouldn't take any rating metric seriously unless it accounted for how long they've owned it. Plus there's the difference between "I've owned 12 lawnmowers and this one's pretty okay, only broke down after 1.5yrs, 3 stars" and "this is my first lawnmower, used it once, it's amazing, never going back to cutting my lawn with scissors, 5 stars". Plus niche sites sometimes have better reviews for the same products (newegg).

All that said, there is information to be harvested in some of these reviews.

Good point about niche sites and the length of ownership. I've noticed that a number of starts on products like top rated power supplies goes down after a year an a half or so. While you might be getting a newly released power supply with 500 reviews averaging closer to 5 stars, after 2 years, you will notice that same product with 1600 reviews averaging at below 4 stars, because owners probably came back and updated their reviews.

It would be interesting to see a trend line of change in sentiment about a products over time, using existing data.

The other common pattern being shill reviews:

   5 stars: ||||||||||||
   4 stars: ||
   3 stars: ||||
   2 stars: |||||
   1 star:  ||||||||

> most people think in absolutes: Either something is the bee's knees or it's fried shit on a biscuit. (Yes, this is from two levels up.)

There was a fascinating post to the OKCupid blog discussing the distribution of attractiveness ratings (also on a 1-5 scale).

Ratings of women by men looked like this:

    5: ||||
    4: ||||||||
    3: ||||||||||
    2: ||||||||
    1: ||||

Ratings of men by women looked like this:

    5: ||
    4: ||
    3: |||
    2: |||||||
    1: ||||||||||||||||||||

EDIT: As you might expect when working from memory, I got some details wrong. Post is here: http://blog.okcupid.com/index.php/your-looks-and-online-dati...

Obvious corrections:

Scale is 0-5, not 1-5

Curve for women is not as steep as depicted in my comment. However, the modal rating is 1, followed by 0. In the words of the post, "women rate an incredible 80% of guys as worse-looking than medium. Very harsh."

A lot of times, the 1-star ratings are only "knee-jerk reactions". The 5-star ratings are sometimes added by people related to the product (or people who for some reason have an agenda). I usually drop those two and focus on the 2-4 reviews.

Who rates something 3? Why bother to interact with a UI to express utter meh?

Reply on news.ycombinator.com