Hacker News Reader

Comment by tensor | Hacker News Reader

One idea to improve the state of things is to require graduate students to verify some number of external studies. In addition to helping with the problem of not enough review, it would make an excellent practical test for doctoral candidates.

It wouldn't work for every field and area, but it could work for a significant subset of research.

View on HN

require graduate students to verify some number of external studies.

That's how Reinhart and Rogoff's “Growth in a Time of Debt” spreadsheet error was finally unearthed, unfortunately that was after it had already been instituted internationally as public policy, but hey, you can't have everything.

http://www.dailyfinance.com/on/reinhart-rogoff-debt-GDP-spre...

As a PhD student in CS, this is an incredibly good idea---even if it stops short of verification and stays somewhere at "reproduce". It can be a lot of work to reproduce a single publication, and often requires very careful reading and attention to detail.

I've done it a few times myself (partially because my work requires it and partially because I wanted to convince myself of the work I was using), and it was an incredibly valuable learning experience.

In the case of CS, replication should become a matter of 'git clone' and 'make'. (Or running a prepared virtual machine, etc.) Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult.

Not just in the case of cs. It could work on electronics, where building some stuff is relatively easy, given design data.

In general, a replication requirement might drive us to build replication tools for each science, which could be quite useful.

> Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult.

I don't understand what you're trying to achieve by saying this. Did I somehow imply that people should be excused for not making their work reasonably reproducible?

It my sub-field, it is utterly ridiculous to expect to reproduce work by running `git clone && make` or by getting a prepared VM. Computational biologists are not known for their systems or software skills.

Well, the context here is discussion of the problem of unreplicable research. I do get the impression (including from your post and reply) that computational biologists need to get their act together on that front; see http://ivory.idyll.org/blog/ for one researcher in that field who sometimes blogs on this theme.

Indeed, we do need to get our act together. I'm still just starting, but I have a couple of ideas on proactive things I can do, but it's an uphill battle.

Thanks for that link---it's always nice to see other people in the field concerned with this. There aren't enough of them.

Good luck! I know it's a big task to change the culture (and I didn't mean to imply I thought it'd be easy). You might like http://michaelnielsen.org/blog/reinventing-discovery/ too.

I'm not convinced that's "replication" in the sense that's important in science. Bugs in the original test will be reproduced in the attempted reproduction. It's useful for "this person didn't flat out lie about their results", which is probably also a good thing, and for finding issues when replication fails, so I don't disagree that it should happen. I just disagree with calling it "replication".

Fair enough -- maybe we should find a different word. It's a reasonable bare minimum standard in the computing world, where it's practical: it makes it possible to trace back from questions and problems with the claim.

Sure - it's clearly a good idea, where feasible (and git repo + whatever-special-hardware should allow the same thing, for the one reasonable case I can think of where it wouldn't be feasible).

Motivation is an issue - each verification would require a bunch man-months of effort, so it won't happen unless there is separate funding for that or somehow magically started to be as prestigious as putting the same effort in a new experiment/publication.

"Requiring" has the same motivation problem - those who could require it, currently would rather require those students to do something that brings funding or prestige, so they won't.

Graduate students are already required to take courses and various tests that don't bring funding or prestige. This could simply be an additional requirement, or replace an existing requirement.

In time, such a program would bring quite a lot of prestige as flaws are discovered and fixed in existing work. It's really the easiest and most immediate way to address this problem since it only needs to involve single institutions (whose faculty presumably care deeply about this issue).

It doesn't seem likely that journals will suddenly start valuing verification work. Similarly, politicians and funding agencies appear uninterested in actual science; they care only about their careers or immediate application to the politically popular cause of the day.

Graduate students already verify people's work. Because every new graduate project involves building on the work of previous science. Which by definition involves re-verifying the work to show you get the same results.

If the results are hard to replicate, or dependent on other causes, then usually that's when a project shifts or when the knowledge pool expands (for example chemistry is fraught with environmental effect dangers - fluorescent lights provide UV to reactions, your glassware has imperfections, the temperature and humidity of labs varies with climate).

I would say that falsifying an existing publication should easily be at least as prestigious and funding-worthy as having made that publication in the first place.

This is an excellent idea for a graduate course; it could be more collaborative for bigger projects, or one-offs for smaller ones, dependent on the field and type of study, as you've mentioned.

I think students should get course credit for it, instead of adding another requirement for PhDs, just to keep them from taking longer than they already do...

I think this is a horrible idea. The average age of R01 has risen to over 40! Why add another 2 years by requiring students to reproduce work that is already published? It adds virtually no value to a graduate degree. The most difficult part of science is creating new hypotheses and designing and executing the experiments to test them. It takes years to learn how to do this as is, without adding on time to use a method you have no interest in, to re-demonstrate an experiment that has already been designed, to test a hypothesis that is already in the literature.

I like the idea. Though in my field (mathematics), I think this is actually not at all uncommon already (reviewing and sanity checking unpublished work of your adviser and their colleagues).

Then again, we have the benefit of being able to "make" our own data so to speak, don't have IRBs to worry about, etc. This is trickier in other disciplines.

Reply on news.ycombinator.com