A Stylometric Inquiry into Hyperpartisan and Fake News
arxiv.org
arxiv.org
manually fact-checked by professional
journalists from BuzzFeed.
That said, and anecdotally, style is often a fair heuristic for political agenda. It should be possible to improve the methodology of such research so it is insightful without such arbiters.
As an aside, it is not surprising that "that the style of left-wing and right-wing news have a lot more in common than any of the two have with the mainstream", for they often use the same, or similar tools to undermine society, a goal both have in common. They read the same literature about how do this, typically following a Lenisist approach.
As an aside, I once developed software for an old telecoms protocol, and found a large number descriptions of this protocol in the trade press. They were all terrible, but felt quite similar. Eventually I managed to track down the origin: teaching material for a course developed by Cisco. As far as I can tell, not one article in the trade press was original. Yet, clearly some authors did quite a bit of work to disguise their copying, including -- bizarrely -- to redraw (badly) the figures in the original material.
Most publications, fringe or not, have journalistic style that would, in my opinion, make them hard to distinguish. That is, if you took a Brietbart article and a Vox article on two different issues and compared the format/length/general tone and language, they wouldn't be all that different.
However, I think these outlets do vary in the specific rhetoric they use, the people they rely on for comment/as sources, etc. I would therefore focus on specific terms or language rather than general style.
2. Buzzfeed, and a number of other traditionally disreputable sites, have really stepped up their journalism game over the last year or so. If all you know about them is listicles, you might be surprised at some of the stuff they turn out nowadays.
I would not be familiar with (2) as I gave up on BF a long time ago.
https://www.buzzfeed.com/craigsilverman/partisan-fb-pages-an...
Which of the journalists there do you think is just the worst? I mean, they work for Buzzfeed right?
Also, Buzzfeed, while sometimes capable of strong journalism, is not the organization I'd pick to help with this. Why not Politifact, Reuters, 538, etc.?
Critical thinking can go a long way to identifying fake news. Our brains are pretty good at detecting bullshit if we filter out our outrage. Though, it might take a little bit of effort in digging through news. A better solution might be to identify articles that oppose one another and pin them together.
It's really not hard for professional journalists to fact-check partizan articles, and I'd assume the ones hired by Buzzfeed used several well-known fact-checking sites, including Politifact.
> sounds like a way to abrogate the responsibility of the individual to understand and analyze sources
But of course it doesn't do that. It's just a way of getting a machine to identify partizan stories that are statistically likely to contain a higher proportion of lies, especially if they are from right-wing sources (Table 1).
As we know all too well, individuals commonly abrogate the responsibility to understand and analyze sources all by themselves.
I think if a system like this got traction, then like many things academics produce, it will be misconstrued and misused.
> It's really not hard for professional journalists to fact-check partizan articles, and I'd assume the ones hired by Buzzfeed used several well-known fact-checking sites, including Politifact.
You should not assume things like that. And again, if they were just gonna look stuff up on Politifact, why couldn't the researcher do that themselves? It's clear there was a more in-depth process here. I think it is totally fair to say that the process could be a good one even if Buzzfeed is a bad choice of organization to run the process with, but further research would have to bear that out. Because only Buzzfeed was involved, it's impossible for us to determine whether there was a bias in the data because of who produced it.
That is, we can't be sure if that data would be similar if we had, say, the National Review doing fact-checking. That's why we need either (preferably) more than one fact-checking organization, or at least a single but highly neutral one.
> As we know all too well, individuals commonly abrogate the responsibility to understand and analyze sources all by themselves.
Which does not mean we shouldn't be careful about enabling it.
My fear here is the common trope of "X new discovery is an oracle," where we take some interesting tidbit about patterns (in in this case, in linguistics) and decide it is absolute truth or applies everywhere.
On the contrary, this is a very reasonable assumption. You appear to be confusing opinions and facts.
You also appear to have an uninformed opinion of Buzzfeed News and the journalists it employs.
You also don't seem to understand the main point of the whole paper, which is that "the writing styles of otherwise opposing orientations, namely left-wing and right-wing, are in fact very similar: there appears to be a common style of extremism."
Based on my long experience as a newspaper journalist, this was already obvious to me and no doubt many other people. However, it won't make any difference to the significant proportion of the American public that doesn't read news for honest information, but only to prop up their existing and highly distorted views.
It's not reasonable to imagine a study used methods just because you would have done so if it were you, even if you know that those methods are the best or most standard ones, then defend the study on the basis of the imaginary method. The point of having a peer-reviewed study is that it clearly indicates how the fact-checking was done.
> You also appear to have an uninformed opinion of Buzzfeed News and the journalists it employs.
On the contrary - I read them and am quite familiar with their work. You can look through my comment history and see me discussing them quite extensively just the other day. Generally speaking, their journalism work is excellent, but it does have somewhat partisan viewpoint in my opinion and they do not always act as an organization in ways that I support.
But again, even if Buzzfeed is not the worst choice, it certainly isn't the best one; you really aren't addressing the point I've brought up a couple times now that there are other organizations better suited to this particular kind of journalistic work.
> You also don't seem to understand the main point of the whole paper, which is that "the writing styles of otherwise opposing orientations, namely left-wing and right-wing, are in fact very similar: there appears to be a common style of extremism."
I understand the point. I just don't think it is useful, and even if it is, I don't particularly think the study has proven it beyond doubt.
Fair enough: I didn't look at your comment history...
> but it does have somewhat partisan viewpoint in my opinion
There is no view without a point of view, and all you're saying is that your own view is to he right of Buzzfeed's.
Meanwhile, individual journalists at Buzzfeed News also have views that are to the right and to the left of Buzzfeed's.
None of which has anything to do with the actual paper, which wasn't about views but about facts.
Actually, my personal political beliefs are probably substantially to the left of Buzzfeed's. That doesn't make me incapable of identifying a partisan viewpoint. I can still tell, even if I mostly agree with it.
> None of which has anything to do with the actual paper, which wasn't about views but about facts.
It has everything to do with it. Partisan sources will rate other partisan sources on their side as more truthful than partisan sources on the other side. One possible explanation of table 1 (not necessarily the true one, but a possible one) is that the reason right-leaning sources were found to lie so often is because they said things the left-leaning fact-checkers didn't agree with, or that the left-leaning fact checkers were harder on the right-leaning sources than they were on the left.
Using Buzzfeed opens the study up to criticism and may detract from the value of the data as a result.
But you're welcome to quote me an example from, say, Politifact or the Washington Post fact-checker or Snopes....
So, anything from the right should be a lie by default? Then google or whoever can just unrank the article to oblivion, since it statistically contains "lies" according to a biased majority? Do you see how dangerous this is?
You must surely know that many right-wing sources are notorious for lying, especially Breitbart and Infowars. Being a right-wing source shouldn't mean your lies get a free pass.
For some interesting information on how right-wing sources are obliged to lie or be rejected by their target audience, see http://uk.businessinsider.com/conservative-media-trump-drudg...
Bias can be dealt with by being an open system. People can see the algorithm, data input, and judge whether the output is fair. Independent audit and review of the system would be miles better than what we have now.
As far as bad-actors trying to game the system, that's another matter.
Having a rating system is just a tool. It will make life easier for every consumer of news.
Life with a spam filter is way better than no filter. We learned this in the 1990's after spending a decade writing them for email, when email became nearly useless as a tool because of spam.
I believe it's important, however, that fake news filters need to be configurable by the end consumer (=me) rather than by someone else such as the site that delivers the news. But that's no different from spam, I'd never use server-side spam filters with unknown blacklists either. If you think you're being tricked, you can occasionally turn off the fake news filter and check the result yourself, or look in your "fake news folder", so I don't see what harm could be done except improving the information quality.
However, I don't think fake news can be detected heuristically by text data mining in the same way as spam, instead the detector must take a look at the data sources and evaluate those.
I understand the metaphor you're going for - and yeah, it's a huge slog to have to comb through all the garbage sometimes. However, there is a pitfall in doing this for too long without re-checking yourself: The trustworthiness of organizations changes, and new ones pop up all the time. If you aren't careful, you can end up in a massive echo chamber.
That is, if you're going to filter out what you feel is "fake news," you'd better have an ironclad definition for the term, and a way to easily tell what does and doesn't match it, and the only way to do that is to be aware as possible of what's out there. After all, how many times a day/week/month do people actually look in their spam folders?
I'd also like to suggest that the problem is vastly exaggerated. Maybe some people really have forgotten crucial distinctions like those between news, false news, fake news, commentaries, opinions, etc., but I doubt that there are many. An almost certain sign of real news is that they are repeated everywhere, since there is only one reality. Anybody who is indeed unsure about whether some news is genuine or not can simply check major newspapers and press agencies, and if they don't take up on the story after a while, then it's not news at all or at least false news.
Maybe in some regions of the world the number of people who are willing to manipulate media with a strong agenda has increased. I'm inclined to believe that, although some hard empirical evidence for this assertion would be desirable. Anyone who uses the term "mainstream media" is very suspect, for example. However, most of these people are really just hypocrites rather than being misinformed.
I can tell fake news from real news. However, I'm left-leaning. And sometimes, there's left-leaning news that is fake. In some cases it's blatant. In others it's subtle. But I have to admit, I'm biased, and I often don't figure out what's true or not until I've done some digging. In some cases, doing some digging results in more fake bullshit.
But that's not really the problem for me. I don't want my world to be a difficult puzzle that I have to solve every day. But that's the way it is. The problem is the minority of people who can't even figure this out. They vote. And they're enough of a minority that this threatens my livelihood and well being, and possibly my life.
This is a huge problem for Democracy, and if we can't reform our electoral system so that a majority wins, then we're actually fucked. Like; we're (USA) going to end up like Egypt or Ukraine fucked.
I think if we can develop tools to combat this, we should use such tools. Or we will become tools for someone else.
However, speaking in general, people are not as good at determining fact from fiction as they think they are. In general, it is a skill that has to be learned, and we live in a constantly changing media environment where one has to keep in practice and remain aware of what's going on, and what's out there. Personally, I know that I have accidentally fallen for things I believed to be legitimate, but that were not, and as a result, I had to improve how I went about verifying news. If I had relied on my own certainty about being able to judge these things, I would have repeated lies.
There's nothing wrong with using an aid day-to-day. The problem only comes when the aid ends up being a crutch, that is, when the point of the filter is to encase you in a bubble, rather than strain out the real crap.
People have little time. A lie reaches many more people than its subsequent retraction, and this is the case even for mainstream news. If the individual has a responsibility to be reasonable and skeptical, distributors of information have a responsibility too. This isn't about censoring stories, but adding fact-check links to articles, the same way you could have a browser extension that tells you if an article has been discussed on HN before. It adds more info and doesn't remove any.
Facebook, Google and others just recognize that an informed population is essential to democracy. If you were trying to weaken or subvert democracy (either as a political radical, or an adversary nation state), you would do it by spreading FUD which sows distrust and conflict.
It's in everybody's interest that the facts be accurately represented in public discourse. Just as the solution is always more speech, not less, here the solution to flawed or biased fact-checking is more fact-checking, not less.
Edit: want to add that fact-checking is journalism's necessary and inevitable response to a rise in non-conglomerate journalism. Before, it was just the radio or newspapers, so they were mostly held to a certain standard. Now, anyone can spread news. You can't have asymmetry like that without a justified response from news orgs.
If people are going online, and reading this stuff and getting all angry and spazzy such that they go buy a gun and shoot up a shopping mall. . . then "source analysis" is actually becoming a serious public safety issue.
(actually more dangerous when the delusional and deranged vote).
Sure: there are three factors at play in these incidents: Availability of guns, (which we apparently can't address, because 'muh second amendment'), Mental Health (which we apparently can't address because 'muh premium's too high'), and Inflammatory Fake News (which we also can't address because 'muh first amendment!').
However, I don't think it's infringing on free speech if a social media site slaps a "source credibility rating" banner on shared agitprop.
https://docs.google.com/spreadsheets/d/1ysnzawW6pDGBEqbXqeYu...
So as far as I'm concerned, so long as the authors acknowledge their source (which they did) then there's nothing wrong here. Even FiveThirtyEight builds most of their models from datasets obtained from third parties. A lot of data science research is carried out using public datasets because collection is often an expensive or time-consuming endeavor, especially for social science research. The authors even say in their acknowledgement "We thank Craig Silverman, Lauren Strapagiel, Hamza Shaban, Ellie Hall, and Jeremy Singer-Vine from BuzzFeed for making their data available, enabling our research".
At first I thought this was an onion article. It turns out they're serious.
Anyone else stopped reading there? Because I did.