Hacker News Reader

Comment by mironathetin | Hacker News Reader

Without the information who financed these numerous research results, they are not worth the time to read them or to write about them, imho.

View on HN

The article also relies heavily on a metaanalysis, not just individual studies. Meta analyses are designed to take a large multitude of studies in the same area and ascertain an effect size. By examining everything in aggregate, this should counteract any funding bias -- or reveal it if the results get skewed (i.e. lots of positive results and lots of negative results but no middling results). Additionally, meta analyses usually consist of a "file drawer" search where they get their hands on a bunch of the unpublished papers and analyze those as well to ensure they counteract any publishing bias for printing papers that only show positive results.

I don't know where you got the idea that metanalysis is mostly unpublished work. I've never seen a published meta analysis like that. Most journals wouldn't print it.

It's happened in some notable cases; for example, a major review of reboxetine, a Pfizer antidepressant, obtained unpublished data from German health regulators and demonstrated that there was no benefit over placebo. The published data gave the opposite impression.

Eyding, D., Lelgemann, M., Grouven, U., Härter, M., Kromp, M., Kaiser, T., et al. (2010). Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ, 341. doi:10.1136/bmj.c4737

But yes, it's not standard practice. Meta-analyses try to check for publication bias but they do not systematically hunt down unpublished results.

I work first hand in a research community and I've never seen a meta analysis published that was based solely on unpublished works. However, most of the good ones consist of a mix of published and unpublished works to ascertain whether there's a real effect or whether there's been a publishing bias that doesn't reflect the total body of research. A good meta analysis will take the time to discuss their methods for selecting published works and any efforts they made to solicit unpublished works such as contacting prominent authors in the field, getting on mailing lists, etc. The good ones also tend to discuss how they analyzed the papers for selection, discuss the paper stats (i.e. we started with 200 published papers and 20 unpublished ones, after winnowing this got reduced to 130 published and 10 unpublished), and how they collapsed results to calculate effect sizes etc.

Meta analyses should include as much of the published literature as possible for the research question they're attempting to answer AND they should seek out unpublished works. The meta analysis will consist primarily of published works, the unpublished works are collected to counter the presence of positive result publishing bias. This provides a fuller picture of the research field than simply relying on published works. After they've collected the articles, the researchers analyze the collected works to ascertain which ones actually address the research question(s) they're attempting to answer, they toss everything else.

You have to view it at an individual study/paper level (ie: go to the link and pdf via the article). They have to declare sources of funding in the papers, so you can trace funding origins and conflicts of interests from those declarations in each study.

The article itself does a nice job of linking to every necessary study - sometimes they don't bother and then you do start to get worried about how much of the content is accurate.

http://en.wikipedia.org/wiki/Ad_hominem

No, that would be calling out the author by name as a shill. Remarking that the source the author is relying upon could possibly be biased/tainted is not an ad hominem in any way.

It could be considered an ad hominem attack on a study if OP had said "the source was funded by coffee sellers and therefore it's all false and incorrect". But that's not what's happening here anyway.

The OP is "implying" that based on who funded the research, this may be truthful. So while the claim was not made outright, it was insinuated.

If the strength of an argument depends partly on trusting the integrity of a source, why is doubting that integrity a fallacy?

In an ideal world you judge experimental results by their reproducibility, but who on HN has the time to reproduce results?

If all you are doing is casting reasonable doubt, that's fine. The logical fallacy arises when you point to the source of a proposition and declare that mouth that spoke it makes it false by nature. There's a big difference between "This pro-coffee study was funded by Big Coffee, so I'm skeptical and would like further studies and independent verification" and "This pro-coffee study was funded by Big Coffee, therefore the conclusion is false and coffee actually isn't good for you." Even if Big Coffee funded this report and benefited from its conclusions, the conclusions may very well still be valid; it would be a fallacy to reject this possibility outright.

Careful, OP isn't claiming that a lack of funding data makes the article false or inaccurate. The point is simply that (in OP's opinion) some of the most critical information needed to determine whether or not this article's conclusions should be taken seriously is missing. This is not an instance of an ad hominem fallacy.

I agree it wasn't a perfect fit - I'm not fluent enough in debate terminology to know, maybe this? http://en.wikipedia.org/wiki/Association_fallacy

I don't think there was a fallacy involved at all here. If in fact critical data is missing, then OP is right to point that out. This doesn't inherently negate the conclusions of the article (and claiming that it does would in fact be a fallacy), but I don't think OP was claiming that anyway.

Reply on news.ycombinator.com