Hacker News Reader

Comment by simonsarris | Hacker News Reader

A great method for spotting (or at least suspecting) fake data is to see if it follows Benford's law. (So remember to fit your fake data to conform!)

http://en.wikipedia.org/wiki/Benford's_law

> Benford's Law, also called the First-Digit Law, refers to the frequency distribution of digits in many (but not all) real-life sources of data. In this distribution, the number 1 occurs as the leading digit about 30% of the time, while larger numbers occur in that position less frequently: 9 as the first digit less than 5% of the time. Benford's Law also concerns the expected distribution for digits beyond the first, which approach a uniform distribution.

As one might imagine:

> A test of regression coefficients in published papers showed agreement with Benford's law. As a comparison group subjects were asked to fabricate statistical estimates. The fabricated results failed to obey Benford's law.

So keep that in mind, data fabricators!

View on HN

Just to explain this a bit, it has to do with relative growth/shrinkage and the base of the positional-numbering system you're using. If you have a random starting value (X) multiplied by a second random factor (Y), most of the time the result will start with a one.

You're basically throwing darts at logarithmic graph paper! The area covered by squares which "start with 1" is larger than the area covered by square which "start with 9".

> You're basically throwing darts at logarithmic graph paper! The area covered by squares which "start with 1" is larger than the area covered by square which "start with 9".

Best explanation for Benford's Law I've ever seen. I've always had a rough understanding of the "why" of it, but this really drives the point home.

You have to be careful with this though. Benford's law only holds for certain types of data, such as growing populations. It has to do with the nature of the decimal system.

To be more specific, it has to do with the nature of any positional-notation system, whether it is decimal, octal, hex, or even binary.

Wow, thinking about Benford's Law in binary is really interesting, thanks for this comment.

It's the most extreme expression of the concept. Every number in binary starts with 1. The only other possible leading digit would be a meaningless leading zero, so every number starts with 1. Benford's Law holds in the most extreme manner possible in binary notation, that numbers starting with 1 dominate the population over all others.

It has more to do with the fact that a good chunk of natural phenomenon are distributed exponentially, not linearly. Benford's Law actually applies to other notational systems.

That only helps you spot outright fabrication though, doesn't it? I would assume that this only happens in the minority of cases.

The much bigger issue is people trying many different statistical tests until they find the one that "works", removing outliers to make the data look better or just neglecting to mention things like p-values or confidence intervals in the hope that nobody would notice.

Much harder to spot, but potentially as misleading as just making numbers up...

most real datasets apparently have outliers... you should not remove them if you want it to look real (I heard this from this course: https://class.coursera.org/dataanalysis-002/)

(but probably don't fit it exactly)

Reply on news.ycombinator.com