Hacker News Reader

Comment by rubidium | Hacker News Reader

Too bad it employs the PolitiFact database, as that itself is suspect to having its own bias. (http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection...)

Using a biased fact-checker to check facts doesn't really work well. Constructing a non-biased fact-checker is the real challenge here. I'm not sure it can be done, due to the great amount of spin that can be put on most anything political.

EDIT: BTW, I'm not meaning to take sides on this, just pointing out the very real difficulties in establishing a "fact-checker". It's worth knowing that others have contested PolitiFact's methodologies. If the trustworthiness of the statement is based on a person's judgement, then the fact-checker has bias.

View on HN

have you read that article in detail? i have spent some time on it, and am having trouble following the argument. at the start they say that approximately equal numbers of statements are taken from either side. then later they seem to be saying that a bias exists because the republicans lie more than the democrats. what i'm having a hard time understanding is why that is a bias - perhaps some republicans simply lie more?

again: it's not they are focusing more on one party than on another (they don't: "the number of public officials subjected to PolitiFact's Truth-O-Meter lens from each party is fairly even during the period under analysis"), but that they catch more lies from one party (the republicans) than the other. in particular, it appears that palin and bachmann are responsible for raising the average number of lies for the republicans.

so why is this bias? why isn't it simply that palin and bachmann are liars? what am i missing (i am not american, so don't know much of the context - i just read the article and couldn't follow the logic of the argument)?

one possible alternative is that they are ignoring lies by democrats and instead checking only true statements. but in that case you'd think there would be a smoking gun - notorious statements from democrats that weren't checked. the article doesn't give any evidence for that.

ps also, i take issue with your final disclaimer. you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. one way to do this is by looking at what is given (for example, this article) and checking whether it makes sense. maybe i have made a mistake - if so, hopefully someone will correct me. eventually this thread will lend weight to one conclusion or another...

you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others.

I agree with this in the fuzzy world of politics you have no choice but to bring in probabilities. I would think that this system could be automated by software to aid the fact checker in coming to a decision. I would say that aggregating a politicians history of lying and using that as a weight as to whether a suspect statement could be untruthful would help.

If a candidate has a history of being less than truthful and a certainty of fact cannot be deduced one could look at weight factors such as history of truthfulness on a subject to help them come to a conclusion.

These methods and the algorithms could them be published for scrutiny. This way it is not arbitrary, it may not be 100% accurate but at least the system is open and the method reproducible. From there it can be refined to help aid in accuracy.

> so why is this bias? why isn't it simply that palin and bachmann are liars?

I hesitate to get into this, but Peter Orszag has only been tagged by Politifact once[1]. Yet at the time he was CBO director, there were many people stating that CBO was distorting long term projections to show that Obama's health care bill saved money. And recently the administration appeared to acknowledge that these projections were (at a minimum) indeed inaccurate[2,3].

  By scrapping the CLASS Act, HHS is losing about 40 percent 
  of the savings healthcare reform was supposed to generate. 
  The Congressional Budget Office’s most recent estimates 
  projected $210 billion in total deficit reduction from the 
  healthcare law, $86 billion of which would have come from 
  CLASS.

  HHS officials emphasized Friday that the healthcare reform 
  law is still projected to cut the deficit, since about 
  $125 billion of the most recent savings estimates are 
  still intact.

Now, you can determine for yourself whether you believe HHS and CBO's other projections given that this program was found to be insolvent before it was even launched. Please also note that there was no shortage of people vociferously opposing[4] the healthcare bill before passage who called out this particular provision as unsustainable due to adverse selection effects.

  The CLASS Act looks like a gift: It brings in $58 billion 
  in net tax revenues by 2019, lowering the deficit by an 
  equivalent amount because only minor costs will be booked 
  during that period. Under the CBO rules, the CLASS Act 
  technically covers one-quarter of the $250 billion 
  shortfall in funds needed to pay for health-care reform.

  The gimmick lies in looking only at the CBO's ten-year 
  budget window. The extra revenues are an illusion because 
  of the disaster lurking just beyond that horizon.

  In fact, none of the $58 billion is available to pay for 
  the House bill. The CLASS Act is so poorly designed that 
  the $58 billion reserve and all future premiums won't come 
  close to covering the generous benefits it's promising.

  Here's why the mechanics of the CLASS Act assure its 
  eventual collapse.

You may disagree, but this is one relatively cut and dried example in which the critics of the healthcare bill appear to have been proven right. Indeed, evidence-laden, numerical critiques of this particular "CLASS Act" provision were all over the web along with charges of falsehood by Orszag.

Yet at the time when it might have moved the needle, Politifact was silent on this issue. This may be what OP is referring to in terms of a selection issue. They simply did not engage the CBO's projections, even though they were far more amenable to classification as "true" or "false" than standard political statements. They were also inarguably important.

[1] http://www.politifact.com/personalities/peter-orszag/

[2] http://www.nytimes.com/2011/10/15/health/policy/15health.htm...

[3] http://thehill.com/blogs/healthwatch/health-reform-implement...

[4] http://money.cnn.com/2009/09/03/news/economy/health_care_cla...

Is there a particular statement from Orszag that you feel Politifact should have addressed?

Well, this one is interesting.

http://www.whitehouse.gov/omb/blog/10/03/04/No-Gimmick/

  Recently, a lot of attention has been paid to a claim that 
  this deficit reduction is achieved only through a 
  business-as-usual Washington budget gimmick: paying for 
  just a few years of costs with many more years of savings.

  This charge is simply false—and let’s get the facts 
  straight. 

  ...

  Third, this is not a budget gimmick.  The purpose the 
  tried-and-true gimmick described above is to make a 
  proposal that adds to long-term deficits appear fiscally 
  responsible. But if that were the course we were taking, 
  we would expect to see a large fiscal hole at the end of 
  the first decade and larger and larger deficits in the 
  second decade. Instead, over the long-term, the savings 
  under the President’s plan are expected to grow faster 
  than the costs.  So, when the Congressional Budget Office 
  is done with its scoring, we expect it will find that the 
  President’s plan reduces deficits by roughly $100 billion 
  in the first 10 years and roughly $1 trillion in the 
  decade after that.  In other words, health reform should 
  reduce the deficit by growing amounts over the long-term.

  Put simply: Health reform will reduce the deficit in this 
  decade, and it will reduce the deficit by even more 
  thereafter.  There’s no gimmick in that.

A program which was supposed to save $86 billion is declared insolvent and discontinued before it was put into production, by the people who were intimately involved in designing it. They pushed hard to pass the bill and made statements like Orszag did above.

In the interest of being nonpartisan, I think that history will look on these kinds of statements in the same way we now look on Colin Powell's presentation to the UN in the context of the runup to Iraq. At best the WMDs/savings just weren't there, and it was an honest mistake. At worst, they lied. Somewhere in the middle one might call them guilty of wishful thinking and exaggeration. But whatever you call it, it's hard to continue to maintain that those who claimed the presence of WMD/savings were empirically correct.

So you want PF to retroactively check people's predictions to see if they were correct? That would be interesting to see, but it's clearly not what they do, they address current statements based on current information.

Now was there any reason to believe that something in that statement wasn't correct when he made it? (for example was it not going to be scored by the CBO?) Otherwise I don't see the bias.

> Now was there any reason to believe that something in that statement wasn't correct when he made it?

Yes, at the time Orszag made the statement many commentators (including the CNN link a few posts up) contested the CBO's projections and said they were manipulated, with a particular focus on the CLASS Act. Orszag doubled down just like Powell did before the UN, just like the yellowcake incident.

I happen to think this specific case is fairly cut and dried, but short of Orszag admitting to a "lie" it is impossible to know what was in his mind. (Though it should also be said that Politifact does not precisely distinguish between whether a statement is empirically false and whether the speaker is a 'liar with their pants on fire').

Anyway, this is one example that stuck in my head as a numerical one which was so important beforehand, and so checkable both at the time and today, that even partisans on opposite sides might meet up to compare notes afterwards.

Uh, it looks like that accusation of bias comes down to PolitiFact grading Republicans as wrong more often that Democrats... I don't want to get all political, but there's another potential explanation for that, yes?

(There's also the accusation that they target Republicans more than Democrats, but that's easily explained by their methodology, from that post: "If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." If we posit that Republicans are wrong more often than Democrats, we would expect them to show up more often.)

Yes, but the parsimonious explanation is that neither Democratic politicians nor Republican politicians nor any other kind of politicians are particularly careful with their facts, and which you think are more prone to "lying" has more to do with the listeners than the speakers. The idea that any politician is routinely dealing in truth is laughable... and I don't mean that so much in terms of cynicism per se, but because they really have very little incentive to do the necessary huge amounts of work it would take that would bring them no gain, or even less than no gain inasmuch as it would limit their verbal weaponry.

Just as one example of the sort of thing that makes it difficult for me to take Politifact ratings seriously in 30 seconds of browsing, look at: http://www.politifact.com/truth-o-meter/statements/2011/sep/... Mitt Romney gets a "pants on fire" rating for claiming that Obama has apologized to the world for the US on various occasions. To my eyes, it's a desperate attempt to avoid the accusation by deliberately and stubbornly being as naive as possible, insisting that it wasn't anything remotely resembling an apology ("pants on fire" remember, not just "half truth", which I would actually accept just fine on this matter) because it doesn't contain the word "sorry" and never quite fully says "apology". I call it "naive" because in politics everybody knows that there's more to what's going on that just what is being said; would Politifact expect me to be naive enough that when diplomats say they had the traditional "frank exchange of views" that means the negotiations are going well, because nobody said outright they were going poorly? Of course Obama never said "I apologize for the US", but there's rather more wiggle room for interpretation than giving someone who says there were an apology a "pants on fire" rating.

And to be perfectly clear, I see equally dubious straining-at-gnats attacks from conservatives on liberals as well routinely. The only way these "truth goggles" could actually work is to draw from a wide variety of sources and hope the bias cancels.

no, the parsimonious explanation is that the person caught lying most lies most. really. it sounds trivially obvious because, well, it's trivially obvious.

sure, politics is complicated and everyone bends the rules. but that would give a "background noise" that was the same for everyone. sometimes anyone can make a mistake. sometimes anyone can get away with it. there's no bias there.

if you catch someone who lies more than the others then the simple, obvious explanation is that they lie more than the others. if you want long words, then it's similar to the difference between systematic and stochastic errors.

in this context it's a little ironic to say this, but surely only a liberal who was so completely disconnected from reality that they have no idea of right and wrong would argue otherwise: some people lie more than others; you can tell who they are because they are caught lying more. to argue that there is no right and wrong in people's statements is "political correctness" taken to crazy extremes.

of course, there are exceptions: if you only check one person then it's unfair. but as far as i can tell (see my other comment) that is not happening here.

The person caught lying most is probably the person who is the worst at allaying suspicion. PolitiFact's fatal flaw is that they only fact-check a subset of statements: those which sound suspicious.

Further, it's important to distinguish lying from being factually incorrect. A lie implies deceit, being incorrect does not.

In both this and the parent comment, is parsimonious really the adjective you were looking for? In what way is either of these a parsimonious explanation?

Simple, perhaps. Reductionist or nominal, certainly. But the cheaper, stingy explanation? I'm not so sure that was correct.

I've got a strong (but untrained) interest in the psychology of bias, and I don't think it is possible. Humans, even smart humans, even smart humans who study bias are notoriously unaware of their own biases and self-deceptions. We tune out information that challenges our belief system, we form alliances, we make the fundamental attribution error, and we attach meanings that tend to reinforce our beliefs.

It's really good to try to be unbiased, but understand that it's an asymptotic kinda of thing.

You are absolutely right -- and I'm not coming into this project with the illusion that any given source will be universally trusted (or trusted at all). I don't want to create a tool that people will look at and say "all righty then! Time to believe anything this tells me!" In fact, that would be counterproductive.

The real goal is a tool that will trigger critical thinking when it is most needed -- flag pieces of content that are worth spending an extra few seconds / minutes / etc looking into and then help them look into it. I want people to think carefully before infusing controversial information into their world view.

(btw I'm Dan in case you didn't pick up on that)

Very valid points. I think that "facts" in the political dialogue are an illusion and politics has always been interpreting reality. Of cause some "truth machine" could spot misconceptions of obvious facts (e.g. how many states are in the Union) but with anything slightly more sophisticated it is going to be a subjective mine field. Just take some basic question and get studies drawn up by different ideological thinktanks and I am sure they are able to produce the most beautiful statistics arguing that their ideological position always has been and always will be the right one. Especially since PolitiFact attempts to "rate" gray areas beyond just true and false their neutrality will, and has to be questioned.

Reply on news.ycombinator.com