Comment by craftsman | Hacker News Reader

Comment by craftsman | Hacker News Reader

Parent Full thread

I think what the other poster is trying to say is something like the following:

There was a team / teams who planned the mission, designed and built the spacecraft and its systems & instruments, designed and performed tests on the samples, analyzed the data, and published a paper with some claims. The paper outlines their methods, test results, data, and conclusions.

Now, is it okay to question anything in the paper? Yes, absolutely.

How probable is it that something in the paper is wrong or could be improved? Definitely non-zero, it happens a lot. People who understand various aspects of the instruments used, the tests, data analysis, etc, often find some minor or major flaw in scientific papers. Could it happen with this one? Yes, absolutely.

How probable is it that someone on HN--knowing that there are lots of people on HN with diverse expertise--will find something wrong with the paper, and that the substance of the flaw they found is:

> I seriously doubt that it could be "ruled out" just like that.

> For one, the spacecraft (or the empty capsules) could have been pre-contaminated before their launch.

> Second, the capsules could have been contaminated after arrival even if they were "sealed" (perhaps the seal had issues, or maybe it happened when they opened them, since one assumes they couldn't test still sealed capsules, they'd have to open them).

Non-zero, but very low. Could it happen? Yes, it could very well be that the team(s) involved did not consider that contamination could occur before or after the sample was obtained and analyzed. How likely is it? Hard to say, but if it happened, the critique would not take the form of "Hey, it could have been contaminated!" Rather, it would take the form of "I notice in the sample handling procedure that such-and-such technique was applied. Further, in the data analysis, such-and-such test was not done, therefore one could not rule out ..."

You get the idea.

My claim wasn't that "the team(s) involved did not consider that contamination could occur before or after the sample was obtained and analyzed".

I merely gave a trivial (even exhaustive!) enumeration of possible contamination cases that could still have happened (regardless of whether they are targeted against in their protocols), not some novel avenue for contamination that I claim to be "smarter than the scientists" to come up with.

Of course they knew of those contamination possibilities. I didn't say anything about them having missed such basic contamination cases.

What I did (and am) suggesting is that they probably are too confident in their protocols having been executed correctly and being 100% tight with no possibility of error. And that that those avenues of contamination are still a real possibility (protocols to prevent them be damned).

Now, this might not be the case. This might indeed be space travelling RNA-combound and B3 vitamin.

But I'd say that the probability of contamination being the case and someone (on HN or wherever) calling it correctly, is equal to the probability of the samples being contaminated. Which is not that small.

Spotting some novel contamination route they haven't thought off would indeed be very rare and a major feat. But merely been less confident (that they are) in their thoroughness and in their preclusion of contamination, is much easier to claim and be correct while doing so.

> But merely been less confident (that they are) in their thoroughness and in their preclusion of contamination, is much easier to claim and be correct while doing so.

But it's a trivial observation, and one that isn't actionable or testable. For the team to have designed and performed the instruments, experiments, data collection, and analysis to say "We're confident in our results and we have ruled out contamination" is substantive and is backed by data and actual experiments. For someone else to say, "I'm not so sure, I think it could have been contaminated," and then provide no actual argument or data on what could have gone wrong, is probably not a novel observation or comment. In addition, it's an easy thing to say (as you correctly point out) without putting oneself on the hook to show how or why.

Here's what I mean: we can confidently say about every paper ever published that "I'm not so sure." Here are the possible outcomes: (a) the paper stands the test of time, the result holds, and nobody remembers anyone who said "I'm not so sure," or (b) someone finds a flaw in the paper, works with the authors and they publish a retraction or revised paper, and then whoever said "I'm not so sure," can easily (as you say) point out how "they called it," all without doing any actual work.

I'd suggest reading every paper critically, realizing (as most people do) that the results are tentative, and either letting it go at that, or dig in and do the hard work to figure out what may be flawed in the work presented. Then work with the authors or publish a paper of one's own pointing out the flaw and the new resulting analysis.

>But it's a trivial observation, and one that isn't actionable or testable

Yes. But if they make a bold claim that "contamination can be ruled out", it's worth it to say to folks ready to take them for their word that, "it's still a possibility", even if it's not a testable finding but a suggestion.

Sure 'we can confidently say about every paper ever published that "I'm not so sure."'. But that doesn't mean we do say it for every paper ever published, or that we say it with equal weight.

The alternative is taking every research announcement at its word, and acting like it's true until it's taken down by another paper. And even then, why'd we trust the second paper refuting the first got it right?

In the end, it amounts to "if you didn't do the work or the review yourself, and din't find any specific actionable issue with it, then you should have no opinion about the probability of it being true or not".

Which is quite problematic. In the end, we often need and should be able to make a summary assessment of the possibility of a paper's conclusions being right or not (as based on our layman experience), even without working with the data, going through it with a fine comb or trying to replicate it, but based on a sense of probability of the conclusion, our knowledge of past assurances in similar statements being ultimately proven wrong, potential conflicts of interest, and so on.

> then you should have no opinion about the probability of it being true or not

Rather, I said when we read papers we should realize that results are tentative. It's fine to have some personal assessment of the probability, but it's not actionable by anyone, and communicating it doesn't assist in improving our collective assessment of what is going on in the world. I'd also suggest that we probably don't, in fact, "often need and should be able to make a summary assessment..."

Reply on news.ycombinator.com