So on one hand, given the nature of statistics, it is necessary to cut down on the noise. On the other hand, it puts the model before reality, and necessarily obscures anything unexpected or contradictory.
It's a matter of evidence. If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. We can continue to explore it, but we shouldn't give a lot of weight to weak evidence (such as likely false positives) in the face of considerably stronger evidence against.
This does, necessarily, obscure certain types of surprising results when the data supporting them is limited. That's OK. Science sometimes converges slowly.
That doesn’t mean that homeopathy is definitely wrong, but it does mean that the signal should be very strong for us to even be considering it to be not wrong. There are also levels to this: I’m pretty sure there are remedies about which we don’t know how they work, but them working would at least not require us to overhaul the laws of physics. That’s still worse than remedies for which we already have a model of how they work in mind, but at least it’s something.
Although you can’t really quantify that, I think such plausibility arguments are vitally important for any research. In fact, since time and money is limited, all researchers (in any field) will always try to limit themselves to what is plausible. Explaining why your model and your hypotheses are plausible is a big part of any scientific work. Plausibility is already a big part of science.
The very idea of plausibility must always go back to evidence; otherwise it's merely wishful thinking.
That's a normative position: "We should require..." And a normative position with an excellent pedigree is still a normative position.
That's not a bad thing, though. Good science requires good judgement. Good hypotheses do not spring wholly formed from logical deduction, there is necessarily an element of informed speculation and guess-making. Evidence must then be interpreted, weighted even, and credibility assigned to third-party results.
So yes, it is very much a matter of ought and ought not.
Think of it as a spectrum. On one end is the pure flake -- always giving too much weight to the newest evidence, no matter how weak, and therefore swinging wildly between different beliefs (I've heard this called "regressive bias"). On the other end is the pure dogmatist -- always giving too much weight to prior evidence, and therefore holding fast to a prior conclusion, ignoring or dismissing even strong contrary evidence ("confirmation bias"). Somewhere in the middle is the proper level of evidential weighting -- giving both old and new evidence the appropriate level of consideration, and therefore changing beliefs exactly as much as is warranted.