[flagged]
Yes, unimaginable that scientists could have made a mistake that a layman could spot, or just said something to create good PR to their mission, even when it's inaccurate.
They also never unwittingly or intentionally BS in their research papers, right? And only a colleague (or a higher level intellect than theirs) could spot a mistake... /s
>Seriously though, do you really think these scientists are making claims, being widely reported, without having made the same conclusions and analysed the situation with all the data, not just that reported through a story on the Internet .. ?
Seriously, similar blunders have happened time and again.
In 1996, scientists at NASA declared that a 6.3-ounce
rock, broken off from a Mars meteorite discovered in
Antarctica in 1984, contained flecks of chemical compounds
— polycyclic aromatic hydrocarbons, magnetite, and iron
sulfide — that suggested the existence of bacteria on the
Red Planet 3.6 billion years ago. "August 7, 1996, could
go down as one of the most important dates in human
history," intoned one newspaper report. But within two
years the theory began to crack.
Traces of amino acids found in the rock, crucial to life,
were also found in the surrounding Antarctic ice. More
damning, other non-Martian rocks — rocks from the moon,
where it is clear life does not exist — showed the same
"evidence" of life. By November 1998 an article in Science
declared "most researchers agree that the case for life on
Mars is shakier than ever."
This particular case aside, you appear to believe that you need to be smarter than X person to be able to spot an error they made. It doesn't take a very high IQ however to see the error in the above idea.
In any case, based on the insults, did you just wanted to lash out on someone to feel better for yourself?
There was a team / teams who planned the mission, designed and built the spacecraft and its systems & instruments, designed and performed tests on the samples, analyzed the data, and published a paper with some claims. The paper outlines their methods, test results, data, and conclusions.
Now, is it okay to question anything in the paper? Yes, absolutely.
How probable is it that something in the paper is wrong or could be improved? Definitely non-zero, it happens a lot. People who understand various aspects of the instruments used, the tests, data analysis, etc, often find some minor or major flaw in scientific papers. Could it happen with this one? Yes, absolutely.
How probable is it that someone on HN--knowing that there are lots of people on HN with diverse expertise--will find something wrong with the paper, and that the substance of the flaw they found is:
> I seriously doubt that it could be "ruled out" just like that.
> For one, the spacecraft (or the empty capsules) could have been pre-contaminated before their launch.
> Second, the capsules could have been contaminated after arrival even if they were "sealed" (perhaps the seal had issues, or maybe it happened when they opened them, since one assumes they couldn't test still sealed capsules, they'd have to open them).
Non-zero, but very low. Could it happen? Yes, it could very well be that the team(s) involved did not consider that contamination could occur before or after the sample was obtained and analyzed. How likely is it? Hard to say, but if it happened, the critique would not take the form of "Hey, it could have been contaminated!" Rather, it would take the form of "I notice in the sample handling procedure that such-and-such technique was applied. Further, in the data analysis, such-and-such test was not done, therefore one could not rule out ..."
You get the idea.
I merely gave a trivial (even exhaustive!) enumeration of possible contamination cases that could still have happened (regardless of whether they are targeted against in their protocols), not some novel avenue for contamination that I claim to be "smarter than the scientists" to come up with.
Of course they knew of those contamination possibilities. I didn't say anything about them having missed such basic contamination cases.
What I did (and am) suggesting is that they probably are too confident in their protocols having been executed correctly and being 100% tight with no possibility of error. And that that those avenues of contamination are still a real possibility (protocols to prevent them be damned).
Now, this might not be the case. This might indeed be space travelling RNA-combound and B3 vitamin.
But I'd say that the probability of contamination being the case and someone (on HN or wherever) calling it correctly, is equal to the probability of the samples being contaminated. Which is not that small.
Spotting some novel contamination route they haven't thought off would indeed be very rare and a major feat. But merely been less confident (that they are) in their thoroughness and in their preclusion of contamination, is much easier to claim and be correct while doing so.
But it's a trivial observation, and one that isn't actionable or testable. For the team to have designed and performed the instruments, experiments, data collection, and analysis to say "We're confident in our results and we have ruled out contamination" is substantive and is backed by data and actual experiments. For someone else to say, "I'm not so sure, I think it could have been contaminated," and then provide no actual argument or data on what could have gone wrong, is probably not a novel observation or comment. In addition, it's an easy thing to say (as you correctly point out) without putting oneself on the hook to show how or why.
Here's what I mean: we can confidently say about every paper ever published that "I'm not so sure." Here are the possible outcomes: (a) the paper stands the test of time, the result holds, and nobody remembers anyone who said "I'm not so sure," or (b) someone finds a flaw in the paper, works with the authors and they publish a retraction or revised paper, and then whoever said "I'm not so sure," can easily (as you say) point out how "they called it," all without doing any actual work.
I'd suggest reading every paper critically, realizing (as most people do) that the results are tentative, and either letting it go at that, or dig in and do the hard work to figure out what may be flawed in the work presented. Then work with the authors or publish a paper of one's own pointing out the flaw and the new resulting analysis.
Yes. But if they make a bold claim that "contamination can be ruled out", it's worth it to say to folks ready to take them for their word that, "it's still a possibility", even if it's not a testable finding but a suggestion.
Sure 'we can confidently say about every paper ever published that "I'm not so sure."'. But that doesn't mean we do say it for every paper ever published, or that we say it with equal weight.
The alternative is taking every research announcement at its word, and acting like it's true until it's taken down by another paper. And even then, why'd we trust the second paper refuting the first got it right?
In the end, it amounts to "if you didn't do the work or the review yourself, and din't find any specific actionable issue with it, then you should have no opinion about the probability of it being true or not".
Which is quite problematic. In the end, we often need and should be able to make a summary assessment of the possibility of a paper's conclusions being right or not (as based on our layman experience), even without working with the data, going through it with a fine comb or trying to replicate it, but based on a sense of probability of the conclusion, our knowledge of past assurances in similar statements being ultimately proven wrong, potential conflicts of interest, and so on.
Rather, I said when we read papers we should realize that results are tentative. It's fine to have some personal assessment of the probability, but it's not actionable by anyone, and communicating it doesn't assist in improving our collective assessment of what is going on in the world. I'd also suggest that we probably don't, in fact, "often need and should be able to make a summary assessment..."
Things like this do happen, and more frequently than one might expect.
I also didn't take a stance on this research. I've yet to read the paper, and I don't read "science journalism" like TFA. But it's not like the findings are earth shattering given previous research. Therefore I lean towards optimism.
I am simply saying that scepticism is warranted even if a paper is peer-reviewed and published in Nature. The "scientific process" doesn't magically end at peer review. The process goes on, and it pretty much never ends.
If your post was to point out a specific flaw, e.g., that the sample could have been pre- or post-contaminated, then it falls short of anything substantive and actionable. Nobody can do anything with it. Yes, you are correct. But no, there is not anything anyone can do with the critique. If you're going to point out a flaw, it needs to be something that you, the authors, or other PIs can check, perform, etc.
I first replied to a comment implying that just because research is widely publicised it should be accepted at face value, with a counterexample. That was all.
Then someone replied to that with the absurd implication I was somehow trying to denounce all of science based on that one example. I corrected them.
The only thing I said about this specific research was the following:
>>But it's not like the findings are earth shattering given previous research. Therefore I lean towards optimism.
If you inferred any other comment on the research than that, the source of that is your imagination, not anything I wrote.
> If your post was simply to say, "This paper could have flaws and its conclusions could be incorrect," then yes, almost all of us will agree with you. I'm not sure it's big news, however.
Critique doesn't just exist to be actionable.
It can also exist (as in this case) to remind of a possibility. Especially in this case, where the possibility is precluded.
In fact that was the very intended. Pre/post contamination could still have happened, we can't just exclude it because "protocols were followed". Protocols have been followed in other cases where errors got in the way to.
Perhaps you didn't wanted to see a general comment, but somebody spotting a specific error in a specific step of the actual process?