Benford's Law tests on Wikileaks data | Hacker News Reader

Benford's Law tests on Wikileaks data

drewconway.com

30 pointsagconway17 comments

> Also, as suggested in the comments, I used a chi-square goodness-of-fit test to see if the deviation is statistically significant, but it was not; with a p-value of 0.2303. Meaning we would fail to reject the null hypothesis: the observed data were a good fit for a Benford process. That said, the p-value is not so large as to suggest total adherence.

Welcome to stats 101. If you get p > 0.05, it means your test failed to tell you anything. Which means your test failed to tell you about the likelihood of tampering.

You are on the right track, but no, it is more subtle than that. The test could still be meaningful depending on what probability of false positives you are willing to accept.

[1] http://en.wikipedia.org/wiki/P-value

[2] http://en.wikipedia.org/wiki/Type_I_and_type_II_errors

this is a statistical blog post written by someone who has no idea how to do statistics. and even when told how to do it properly by a commenter, fails to accept the simple and overwhelming conclusion. why is this on the front page?

Perhaps because the data is interesting and we can draw the contrary conclusion: that there is no reason to suppose the data has been tampered with or fabricated?

They should expect a bias, since Wikileaks have stated that some information has been held back deliberately. From http://wikileaks.org/wiki/Afghan_War_Diary,_2004-2010

"We have delayed the release of some 15,000 reports from the total archive as part of a harm minimization process demanded by our source. After further review, these reports will be released, with occasional redactions, and eventually in full, as the security situation in Afghanistan permits."

I might be missing something, but how is it that overall the number of leading 2s is greater than the number of leading 1s, but not in any of the constituent regions?

It might be a form of http://en.wikipedia.org/wiki/Simpsons_paradox

I don't think Simpsons paradox plays any role here, if I understand this correctly.

In the first graph, the leading digit counts are not the things being summed up across all the regions. The weekly report counts are summed up across all regions and then the leading digits are counted.

Could it be that the mean number of reports in a week is in the 2.5-3.5x10^x range?

Why would he look at the leading number instead of the trailing number?

He really ought to provide the exact dataset that he's using to generate the graphs.

Benford's law is about the distribution of the leading digit, not the trailing ones.

Right, but was his objective to use Benford's law, or was it to test whether the data was tampered with?

His objective was to use Benford's law to test whether the data as tampered with. (This is a standard use of Benford's law.)

I thought that wikileaks eliminates certain information that would cause harm to a party mentioned in the leak?

That is correct, there is a subset of ~15,000 records that are being withheld, which will be released when they no longer have a probability of being harmful to those they mention.

Source: http://boingboing.net/2010/07/26/wikileaks-qa-with-ja.html

Boing Boing: We're told that there are more documents from this archive yet to be released by Wikileaks, some 15,000 of them as reported. Some have speculated that these could relate to Iraq. Can you comment more?

Jacob Appelbaum: The 15,000 documents are part of the set of Afghanistan documents. They are being redacted for harm-minimizing purposes as requested by our source, and will be made available as is applicable with respect to the relevant security concerns.

I would like to know much more about the process by which Wikileaks determines what is or is not safe/relevant to be released. While I support their efforts, they are walking a thin line in becoming the very thing they are working against.

I only hope that the decision was done by experienced committee, but even that raises questions as to how do you vet that committee for the ability to gauge what is and is not releasable. How does one determine those people are even of adequate credentials/authority to read the information ahead of time?

If I were our the US govt., I certainly would be playing up that issue as much as possible. On the surface it would not be hard to push general sentiment in the direction of Wikileaks playing information Gods with the data they have.

Incredibly weak evidence.

It's not weak evidence. There's no evidence. The guy doesn't seem to know how significance testing works.

An aptly named blog for the analysis.

Reply on news.ycombinator.com