How Researchers Realized Research 2000's Daily Kos Data Looked Faked | Hacker News Reader

How Researchers Realized Research 2000's Daily Kos Data Looked Faked

techdirt.com

95 pointsyanw24 comments

The original report is here: http://www.dailykos.com/story/2010/6/29/880179/-Research-200...

What I find most interesting about this is that the statistics used here really aren't complicated at all: the missing zeros are a pretty classic case of the human vision of random not matching up with reality, and the other issues they found are also pretty easy to understand at a high level. They go (a bit) deeper to show exactly why the problems they found are problems, but it's quite easily apparent that something is wrong when you look at the raw data. Problems in plain sight indeed.

For more on this sort of thing, check out 538's incredibly comprehensive takedown of Strategic Vision last September:

http://www.fivethirtyeight.com/2009/09/strategic-vision-poll...

http://www.fivethirtyeight.com/2009/09/comparison-study-unus...

Yeah I'm not sure how TechDirt didn't mention Nate. He started the whole thing.

What actually bugs me somewhat more about this case is that it seems they were only caught (if indeed it was faked) because they faked it badly. But what if a pollster fakes their polls more intelligently, such that they look legit to at least most basic statistical tests? In well-polled areas, especially (e.g. presidential races), it seems like it would be difficult to distinguish between a real pollster, and a fake pollster who took a bunch of other polls and ran them through a "generate random-looking result roughly similar to these other polls" function. As long as your distribution looks right, and your absolute numbers are roughly in line with the pack, it seems like it'd be hard to prove that you didn't actually robocall anyone.

Also, my understanding is that polls on Kos included more data than is typically provided. Which I think helped this analysis. So a more traditional poll could be faked just as badly and not be susceptable to being caught.

The insight was in knowing to look at differences, week(n) - week(n-1). That was not the raw data.

They also noticed that the percentages of "men in favor of Obama" and "women in favor of Obama" did not tend to be odd/even or even/odd numbers. They were even/even or odd/odd. That's obvious only in hindsight.

These polls are inaccurate even if the data is not faked. If you are doing the poll for Fox News you ask "Are you pleased with they way Obama is completing ALL of his campaign promises?" if you are doing the poll for MSNBC you ask "Given the state of the country Obama inherited are you pleased with his progress so far?"

Not true. If you had been following Nate Silver's site http://fivethirtyeight.com/ in the lead up to the presidential election, you'd have gotten VERY accurate poll results.

Nate Silver isn't a pollster. He is a statistician that dissects the polls to find the inaccuracies.

http://www.fivethirtyeight.com/2010/06/pollster-ratings-v40-...

One thing that has always bugged me is how can two polls give different results to effectively identical questions and both claim to have "plus or minus 3%" (or whatever) margin of error, when their results have substantially larger differences than that?

A slight change in wording can change how people think about their answer

Keep in mind that defining a representative sample is also fraught with differences. How you you determine if someone is a likely voter (are they registered? did they vote last time? other or some combination of the above?), for example, or ensure what you believe is a representative sample of different ethnicities?

Because a sample of 200 - 1200 is used and multiplied out to represent the population as a whole, tweaks in those definitions can lead to different results for the same questions.

I definitely second the www.fivethirtyeight.com tips, as well as www.electoral-vote.com for regular outside analysis of polls and discussion of methodological differences among the various polling places.

I understand that - I meant my comment to be more a complaint about bogus/worthless "margins of error".

The margin of error is often meant as one standard deviation. And for normal distributed data, only around 66% of values lie within that.

Your example is not bad but I like this one better http://www.youtube.com/watch?v=2yhN1IDLQjo

Radiolab, an incredibly well polished radio program on nerdy topics, did a show on maths that included Benford's law. Check it out: http://www.wnyc.org/shows/radiolab/episodes/2009/10/09/segme...

This one makes me wonder: "The first thing they noticed was that when R2K did polls that tested how men and women viewed certain politicians or political parties (favorable/unfavorable) there was an odd pattern: if the percentage of men that rated a particular politician favorable or unfavorable was an even number, so was the the percentage of female raters."

How could they produce such results? Even if the results are 100% fake, they certainly did not do it on purpose.

Take men's results and add a random multiply of 2 to produce women's results? Naah.

Either there's a cognitive bias to pick both-even or both-odd numbers when making things up, or they generated random numbers in [0..50] and multiplied by 2.

The first one seems improbable and the second one would never produce odd numbers which I believe was not the case - it produced both odd and even ones, just never both and even in a single sample.

True.

I'm guessing they picked an integer A for the average of the men's and women's percentages, and then a smaller (positive or negative) integer D for the deviation from this average for each sex. Then they could use A+D as the score for the women and A-D as the score for the men. If one of these is even (or odd) then so must the other be.

It might be worth checking to see the distribution of D, compared to real variation between men and women.

Perhaps A was even derived from some real results? And they just made up A for the cross tabs?

Interesting. Reminds me of my rant against a statistical analysis of the Iranian election results some time back: http://blog.jgc.org/2009/06/scaccobeber-analysis-of-iranian....

As an ex-DSP wonk, it vaguely looks like a quantisation error in R2K's numerical processes. Vaguely.

Reply on news.ycombinator.com