Comment by drvortex | Hacker News Reader

Comment by drvortex | Hacker News Reader

Parent Full thread

Independent replication does occur when someone furthers the research, usually as a preliminary test of current knowledge before diving into new experiments. I personally took great joy when what I had reported was used and confirmed in the preliminary experiments done by other labs for their further studies.

But there is a deeper philosophical issue with your idea of the scientific method :

Consider what is meant by independent replication. Does it mean that the experiment must be repeated from scratch with different material etc. such that it is an independent test from the previous attempt?

Or does it mean that it has to be repeated by different people as well ?

It raises a conundrum. If people are a factor that must be controlled for by this definition, no replication can ever occur since the exact individuals that performed the experiment will have changed by definition during this 'replication' attempt. The skill and knowledge of people is a factor that is notoriously difficult to objectively quantify. This often results in statements such as "X method/assay/procedure does not work well in our hands ".

On the other hand , if people are not a factor, then replication of the experiment by the same group of people will also qualify as an independent replicate.

>"Or does it mean that it has to be repeated by different people as well ?"

Yes. The purpose is to 1) demonstrate the experimental conditions have been mastered to the point they can be communicated effectively, and 2) demonstrate the phenomenon is stable in the face of any unknown factors specific to a place and time.

1. Effective communication of a protocol has no relevance to its efficacy. So that's a plainly unscientific test of the 'truthiness' of anything.

2. No phenomenon ever, scientific or not, can be guaranteed to be stable in the face of all 'unknown' factors. They are unknown, you cannot make any statement about that.

I understand where you are coming from, but that puts you right in the middle of the philosophical quagmire mentioned in my earlier comment. If you mean 'variation in people performing the experiment' by those 'unknown factors' then essentially you are making the assumption that the people are not a factor that is expected to influence the result. As such, even the same team repeating the experiment would suffice. Anyway, you get the point.

It's a time tested heuristic. If a result cannot be communicated well enough for others to replicate or if it strongly depends upon local conditions (doesn't really matter which reason), we should either focus on something else or figure out why. If there is no independent replication, there is no chance to learn either of the above and no reason to have confidence we know what is going on.

I see no quagmire, it is all very straightforward. I will not believe my own results until others replicate them. Even once is not enough to hang my hat on. What alternative approach do you use to judge whether an observation is worth theorizing about?

Reply on news.ycombinator.com