Should our publication have been prevented simply because nobody else had the resources to replicate?
Should our publication have been prevented simply because nobody else had the resources to replicate?
Well... maybe, yes?
Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.
however, If you follow your logic, you would prevent CERN from publishing the discovery of the Higgs Boson, because nobody else has a particle accelerator powerful enough to detect them. You would prevent the LIGO people from publishing because nobody else has a multi-mile-long interferometer that can detect gravity waves. There are many unique resources that would be prevented from publishing under your idea.
Contrary to what most non-scientists claim about science, reproducibility is not a requirement for discovery science. What is import is scrutiny- ensuring that when people discover something using a unique resource, other people can inspect the methods and the data and decide whether to believe the results.
You're addressing another issue: they built replicability into their scientific method (which is awesome) but it's still within a single logical entity (which happens to be distributed throughout the world).
LIGO went one better and injected fake events to make sure their analysis systems were working.
Did you think about the replicability of your research while you were working on it?
As for thinking about replicability, sure I thought about it. That's why we released all the data we generated and described our methods in detail- so pharma and academics could use the data with confidence to design new drugs that target GPCRs. I also decided that rather than focus on replicability, I would focus on discovery and sharing, and I think that actually is a better use of my time as well as the time of other scientists.
I find this an easy and useful distinction and your publication should NOT be prevented from being published by this measure.
What is the difference between your paper and an Uri Geller "experiment"? Both are extremely hard to duplicate for objective reasons so their results have to be accepted based on reputation. (Imagine someone trying to publish the same thing but using "Startup LLC" instead of "Google".)
It's pretty clear what's different between my paper and a Uri Geller experiment. If you can't see the difference, you're either being obstinate or ignorant. We certainly aren't banking on our reputation. A well-funded startup with enough cash could duplicate what we did on AWS GPUs now. I would be thrilled to review their paper.
By whom? I think you're using as a premise the very matter that is being discussed.
I didn't always believe this, but after spending a bunch of time reading about the numerous times in history when one scientist discovered something, published the results, and were told they were wrong because other people couldn't reproduce their results, only to be shown correct when everybody else improved their methodology (McClintock, Cech, Boyle, Bissell etc).
This ultimately came to a head when some folks critized the lack of repeatability of Mina Bissel's experiments (for those who don't know, she almost singlehandledly created the modern understanding of the extracellular matrix's role in cancer).
She wrote this response, which I originally threw on the floor. http://www.nature.com/news/reproducibility-the-risks-of-the-... After rereading it a few times, and thinking back on my experience, I changed my mind. In fact, Dr Bissel's opinion is shared by nearly every top scientist I worked with at UCSC, UCSF, and UC Berkeley. The reason her opinion has value is that she's proved time again that she can run an experiment in her lab that nobody else in the world is capable of doing, and she takes external postdocs, teaches them how to replicate, and sends them off to the world. In other cases, she debugged other lab's problems for them (a time consuming effort) until they could properly reproduce.
I believe reproducibility is an aspirational goal, but not really a requirement, for really good scientists, in fields where reproduction is extremely hard.
Tom Cech and his grad students discovered that RNA can have enzymatic activity. They proved this (with excellent control experiments eliminating alternative hypotheses) and yet, the community completely denied this for years and reported failed replication, when in fact, the deniers were messing up their experiments because working with RNA was hard. Cech eventually won the Nobel Prize.
Stanley Prusiner: discovered that some diseases are caused by self-replicating proteins. Spent several decades running heroic experiments that nobody could replicate (because they're heroic) before finally some other groups managed to scrape together enough skilled postdocs. He won the Nobel Prize, too.
Barbara McClintock- my personal favorite scientist of all time. She was soundly criticized and isolated for reporting the existence of jumping genes (along with telomeres) and it also took decades for other groups to replicate her work (most of them for lack of interest). Eventually, she was awarded the Nobel Prize, but she also stopped publishing and sharing her work due to the extremely negative response to her extraordinary discoveries.
Mina Bissel went through a similar passage, ultimately becoming the world leader in ECM/cancer studies. She will likely win a Nobel Prize at some point, and I think we should learn to read her papers with a certain level of trust at this point, and expect that her competitors might actually manage to level up enough to be able to replicate her experiments.
Because if a piece of work is important enough. People will find the truth. If it is not important enough.. then it does not matter.
And sometimes it is impossible to solve a difficult problem using the first lab one builds in pursuit of that concept.
In these cases the true type of laboratory needed only comes within focus after completing the first attempt by the book and having it prove inadequate.
Proving things is difficult not only in the laboratory, but in hard science it still usually has to be possible to fully support your findings, especially if you are overturning the status quo in some way. That still doesn't require findings to actually be reproduced in a separate lab if the original researchers can demonstrate satisfactory proof in their own unique lab. This may require outside materials or data sets to be satisfactory. Science is not easy.
I think all kinds of scientists should be supported in all kinds of ways.
Fundamentally, there are simply some scientists without peers, always have been.
For unique breakthroughs this might be a most likely source, so they should be leveraged by capitalists using terms without peer.
Incidentally, Scott Alexander just published an article [1] with a great quote:
"Peer review is a spam filter. Replication is science."
[1] https://slatestarcodex.com/2017/02/27/ssc-journal-club-analy...