Why is this a problem? If the experiment's design is not in conflict with the new findings, why complain?
Why is this a problem? If the experiment's design is not in conflict with the new findings, why complain?
The problem is we're not matching up our statistical analysis to our testing when we look for findings then naively test hypotheses based on those very findings.
Statistics can be true and false at the same time. Depending on interpretation of the data and how you try and 'spin it'.
This is why I'm wary of any study that only shows their statistics, but does not share their testing methods.
Bayes can be hacked by ommitting info, but not by things like this, I believe.
The inference in Bayes only depends on the data, and the flips of other coins doesn't make a difference if independent. Frequentist testing can depend on things like stopping rules and hypothesis tested, which aren't correlated to the actual truth and therefore should have zero effect on inference.
They can cherry pick and only show some flips of that coin, but then they really need to be outright lying or you'll ask why only some flips were reported.
Two-up ("Come in, spinner!") is a coin-flipping gambling game where people bet on whether a punter will come up double-heads or double-tails. The house makes it's money when neither come up five times in a row (with some variations).
This is the reason experiment repeatability should be considered of crirical importance especially for 'gooey' disciplines like medical and psychological research where there is little more than statistics to go with. Was the experiment just mininf random noise for correlation - 'simple', just repeat the experiment. If results can't be repeated it's more likely the original authors just botched the experiment, massaged their data to look how they wanted or just got lucky with a random number sequence.
This is similar to the sharpshooter fallacy. If I shoot at the side of the barn then draw a bullseye wherever I hit I can pretend to be a good shot.
This generates an unrealistically high statistical significance (unrealististically low p-value) because the question should not be, "What are the odds these two things are correlated by chance" but "What are the odds that any of of these things is correlated by chance with this other thing?"
The significance of the result is far below the reported p-value from an analysis that assumes only one thing was investigated. You frequently see incorrect p-values of this kind even when the researchers acknowledge they tested for multiple things.