Criticism of “Research Parasites” Moves Medical Journal in Wrong Direction
statnews.com
statnews.com
But the reality is that both the career advancement and funding for further experiments is almost entirely based on the number of publications in prestigious journals.
And effort put into collecting data is not by itself publishable: you must add analysis, and very often analysis that successfully confirms a new hypothesis or invalidates previous beliefs to publish. Journals prefer to publish exciting results, and "we ran experiment X, and found no significant effects" is usually rejected.
Fixing this is desperately needed, but so far nobody has come up with a solution that would actually fix this.
A middle ground would be to require data publication after a certain time period. For example, NASA and ESA release original data 6-12 months after collection. Releasing original data together with the paper also works.
Wouldn't rewarding people for `cites' of their dataset be a way to fix this?
Ie if someone uses your data, they have to cite you, and that's just as good as someone citing an analysis paper.
Edit: I'd love to actually, but perhaps I'll wait for the post scarcity, star trek economy to become mainstream.
Scientists have to eat and pay the bills, too. Unless you're really rich, it's tough to do much serious science in your spare time on weekends, paying for equipment from your own pocket. So the idea is that a scientist gets a job that pays him/her to do science. The system rewards scientists who advance knowledge and penalizes those who do not.
How do we determine whom to reward? It is extremely difficult to evaluate most scientists' work unless you are an expert in the same field. So we let peer reviewers evaluate the work, and reward scientists for publications in prestigious peer-reviewed journals.
The problem is that these incentives work against some kinds of sharing.
It's not all bad. A scientist whose work never sees the light of day is fired; no publications means no job. But sharing of ideas apart from those that are necessarily published is something the system tends to punish.
What to do about it? It's easy to propose some radical revision of the system. But we still have to deal with the question of how to evaluate the work of someone whose ideas can only really be understood after a decade of specialized study.
Edit: no, on closer reading, I am incorrect. My bad.
Or shame them academically. You'd be surprised how far that goes.
It would be really worthwhile for the International Committee of Medical Journal Editors to hear your story, as they are proposing that authors should be allowed to withhold data for six months after publication. See my comment elsewhere in this thread: https://news.ycombinator.com/item?id=11019760
If you have access to PubMed Commons, I would post a comment warning others that the authors refused to make the data available despite promising to do so in the paper, and that people should treat the results with caution.
An optimum is often found that favors sharing as little as possible but often "as little as possible" is still quite a lot. Also, how can people label this as unethical when people would not even have gathered the data if they knew beforehand someone would force them into sharing, diminishing the data's return on investment?
I'm also pretty sure the sharing of a complete data set will get you a lot of citations, so there is a financial incentive next to the moral one.
But many clinical trial researchers want to withhold the data even after publication. This is fundamentally at odds with how science works. It means no one can independently evaluate the results.
The paragraph from NEJM's piece that drew the brunt of the scorn was the following:
A second concern held by some is that a new class of research person will emerge — people who had nothing to do with the design and execution of the study but use another group’s data for their own ends, possibly stealing from the research productivity planned by the data gatherers, or even use the data to try to disprove what the original investigators had posited. There is concern among some front-line researchers that the system will be taken over by what some researchers have characterized as “research parasites.”
The meme has already given birth to at least one parody account: https://twitter.com/dataparasite and one rather tongue-in-cheek domain purchase: http://researchparasites.com
How is this a bad thing?
The original investigators - as scientists and seekers of the valid, the factual, the accurate - should welcome competing attempts, especially given that such attempts may reveal a faulty original analysis.
http://retractionwatch.com/2011/08/11/is-it-time-for-a-retra...
If the papers they publish get more scrutiny, they will undoubtedly have to retract even more.
I'm not sure I agree with it, but I think it's possible that's more what they're talking about. Rather than, say, a lab coming back and saying "we analysed the BICEP data carefully and found the results are possibly the result of dust".
1) Defenders of one of the tenets of the Scientific Method--peer review--and the transparency that that concept demands...
2)Those wary of what the current environment implies might befall those who are totally transparent with both research data and results...
"Research Parisites" are being deemed as synonymous with "Patent Trolls"...
It will be interesting to see how this is sorted...competition is fierce for funding...important fundamental principles, and billions of dollars, are stake...
I guess that could be considered a specific type of peer review. Getting useful feedback is always good too, whether related to science or not. Maybe of interest:
http://michaelnielsen.org/blog/three-myths-about-scientific-...
The authors of this editorial should be stood down.
I feel like hidden datasets are a real problem. In fields like medicine where some authority will look at the data eventually there's at least some quality control (debatable as to how much) but there's entire sections of science were it would be rather trivial to fake entire studies without even collecting data...especially since redoing experiments seems to have a bad rep, too.
Pretty broken system overall. I'd love if there was a step in the acceptance of papers that would say...paper accepted under the provision that the data set is made available. You still get "the glory" because the stuff is published and you're the first source on it but now the data is also available.
Additionally, one of the goals of science is being reproducible and transparent. If an experiment is well described and reproducible, additive data sets could be built. Run a similar enough experiment but with another demographic, add to the data set...etc.
Edit: I also don't see why the role of "data gatherer" can't be more prestigious. I mean sure, traditionally you gather data for a reason and want to answer a research question with it and that's what you're judged by. However there's tremendous value in identifying that no good data set exists for some area and then outlining a solid, transparent and scientific process of collecting that data and executing it. I'd call that a valuable paper even if no hypotheses are tested. As long as there's a place where you can publish those papers and make the data available it would probably also be a paper that is good for your career by a metric that seems to matter a lot. You'd potentially get a lot of citations since everyone who conducts analysis based on the data set or extends it etc. would cite you.
I agree wholeheartedly. If datasets are so valuable that lots of people can easily capitalise on them and produce great science, then the creation of those datasets should be rewarded similarly to an extremely valuable paper.
Machine learning seems to be a field that this is going pretty well in, people are publishing their models and datasets more so that I can grab a trained model of a huge image recognition neural net and try it out on my own data.
> As long as there's a place where you can publish those papers and make the data available it would probably also be a paper that is good for your career by a metric that seems to matter a lot.
I'd personally like to see a shift away from requiring a paper to cite a dataset. But I don't think that really alters your point.
(disclaimer, I work for Digital Science which is a parent company of figshare, but this is a personal opinion)
Gathering enough patients to do a study on some particular disease is often difficult and the result of a career long reputation. Doctors usually refer to it as their "recruitment", ie patients addressed to them because they are the reference in this field, something that these "parasite" researchers will get credit for without having to earn it
That's why I believe they are so protective with data.
Just in the last two weeks, the International Committee of Medical Journal Editors proposed new rules[1] about sharing clinical trial data. Commentators think that kind of data sharing is a very good idea.[2] As a reader of medical research who has heard a lot of "war stories" about the medical research process from family members who have observed that process at first hand, it seems to me that having more eyes on each data set is nearly always a good idea.
[1] http://annals.org/article.aspx?articleid=2482115
[2] http://www.statnews.com/pharmalot/2016/01/27/proposal-data-s...
Patient privacy aside, I do agree with you and with the critics of the New England. Science is about reproducable experiments, and in medecine, because of the constraints on the "research material", most studies are on the edges of statistical significance, and with professionals that aren't trained statisticians.
The only mitigant to that is data sharing.
What we need are alternative funding models for academic medicine. With all the negative press and ethical issues from pharma funding the studies, and the NIH cutting their funding, publish or perish is even more important if you want a sliver of a chance to get some of the remaining grant money.
Since simply providing the data can't be measured for tenure or fit into a line item on a CV, academia as a whole assigns it zero value. This then means that simply sharing data is a huge net negative given the high costs of acquiring it.
In academia, it's better to be the primary author of a paper. More primary author publications makes it easier to get funding, keep your job and advance your career. If you share the data that you might have spent a lot of time and money collecting, you will either gain a citation or a non-primary author position. This is worse for your career. I don't like the system. Most academics would prefer to share to have their work verified or even to see what others might find but the incentives are to keep it private so that you can squeeze as many primary papers (or papers from your research group) as possible out of a data set before sharing the paper.
The actual reason is that there is a whole cottage industry based on trading access to data for paper co-authorship. If the researchers were to share their data openly, the person who comes after them would be able to just cite the data without having to beg the researcher for access while promising him/her a sweet co-authorship spot in return.
Is that a problem? That's the way it should be. Requiring that the data be shared does not rule out a requirement that the person that creates the dataset is automatically a coauthor.
Also, a lot of patients end up exploited by docs and companies who sell these very expensive samples taken from patients with rare diseases. The patients don't benefit from that, but the docs and medical institutions do. So while we try to "protect" medical research for the sake of doctors, they are not only using their patients for profit but they are hindering medical research in the name of their profits.
And yes, I worked for a medical lab where I helped identify and sell patient blood and tumor samples with rare forms of blood cancers. I'm not necessarily against selling patient samples where the patient doesn't benefit, but blindly protecting doctors' claims to research data doesn't tell the whole story.
In that research context, it can take teams of 10's to 100's years to establish a usable dataset which can win grant money to both serve patient populations and reinforce the long-term sustainability of a clinical/academic institution.
You bet your ass that data isn't going to be given out. It's not about covering your ass. It's about stewardship of data which has been culled via consent from participants, often of a very sensitive nature, with specific and explicit limits to how it will be used, managed, and shared.
It's a gold mine that was built from years of getting at the forefront of a niche and then racing year after year to stay there.
The gold from the mine isn't used just to line someone's pockets or add lines to a CV. That gold often is used to provide in the treatment of people who would otherwise be left to the whims of state legislatures that play political football with Medicaid and funding for health services for underserved populations.
Also, from my experience, investigators sitting on good datasets often WANT to share them for use since it establishes increased value and bodes well for future grant competitiveness. Here again... bring your credentials since you are being privy to something bordering on sacred. When I see a term like "research parasites," I know who they are talking about, and no, they are not going to be accepted as a collaborator. It might be a lack of any facet necessary to carry that data stewardship at or beyond what the originator has done - and that is not something you easily demonstrate outside of the existing systems.
All this apart from the most basic principle of scientific philosophy, in which earnest, transparent efforts to find out which theories cannot be falsified has the highest priority.
Arguable, but if they don't have to release their data, then very true -- they aren't accountable. By requiring scientists to release their data, we make them accountable for any shenanigans that might otherwise be concealed.
Yes, but made-up data is also valuable. Were this not true, the world of advertising would collapse.
An earnest economic evaluation has to assess the value of data apart from the issue of whether it accurately reflects reality.
Scientific research is supposed to be a public good in the economic sense[0]. If it isn't done openly and transparently then it isn't and it most certainly shouldn't be publicly funded. This manifestly isn't how it works currently - see e.g. world v Elsevier et al - and this is really just a plea to retain the broken status quo.
If researchers want their work to be of benefit solely or primarily to themselves or their paymasters - a violation of the Mertonian norms [1] - they can go work in industry.
Incentives matter, of course and what's at the root of this is that scientists - no matter how much or how often they may insist otherwise - are just ordinary self interested human beings.
[0] https://en.m.wikipedia.org/wiki/Public_good [1] https://en.m.wikipedia.org/wiki/Mertonian_norms
The ICMJE says that authors should have the ability to prevent release of the data for six months after publication. This will prevent researchers from performing independent analyses of a study, until well after its conclusions have been trumpeted in the press. Also, while many papers currently promise data or other materials are "available on request," anyone who has tried to actually get these data knows that this is often just wishful thinking. Only by requiring data availability in an independent repository at publication can journal editors and reviewers ensure that it will be available to other researchers and the public.
The ICMJE is accepting comments on this proposal through April 18:
https://forms.acponline.org/webform/comments-icmje%E2%80%99s...
If you are a patient or a researcher, please comment.
Am I missing something, that redaction is...odd.
I do not read that as the editor calling researchers parasites. Rather, the editorial is exploring different ways of data sharing and collaboration.
See the NEJM editorial that sparked the controvery at http://www.nejm.org/doi/full/10.1056/NEJMe1516564
The blog post appears to go to some effort to misinterpret the editorial.
1. http://www.partiallyderivative.com/news/2016/1/25/episode-38...
Though what the NEMJ authors seem to desire is more akin to a market - I'll trade you this data for the prestige of a co authorship - this is not 'markets broke science', this is 'scientists believe an explicit market structure will maximise their utility'.
These seem to me to be radically different statements, though this may be a matter of perspective.