It's just so frustrating that professional scientists just completely ignore this statistical fact. Why not use use N >= 100 or not at all. It just confuses things in my opinion.
There are multiple reasons why small N is not bad thing.
Firstly, multi year research projects are not silent for years or decades before they deliver final paper. They publish interim research, technical reports, exploratory research or preliminary research to clarify issues and fine tune the direction of research. You do something and you publish it.
Secondly, multiple studies where N is small can be more reliable than one big study with large N. There are errors other than statistical errors.
Thirdly. Effect size is much more important than N. N=1 can win you a Nobel price if the effect size is big enough. Grow a mice that weights 10 kg and N being 1 or very small number is good enough. There are many studies where N is tens of thousands (p < 0.001) but the effect size is so small that the result is not scientifically interesting.
Finally, statistics is not the final arbiter of importance of the study. It has become standard to add it into every publication, no matter how small. But researchers don't focus on that.
---
Everyone wants to participate in the discussion and most people in comment sections know something about statistics but little to nothing about the subject, so you always get the 'correlation is not causation', 'small N' so it's irrelevant talking points.
Generally because bigger N means the study is more complicated logistically ( cost, finding participants, whatever). And if the results are promising you can always get more funding to move on to bigger trials. Nobody is going to pay for you to recruit 1000 participants if you can't show an effect size with a smaller number first.
> I think its pretty obvious that a small N is not statistically significant
Well this totally depends on the effect size. You can't just dismiss results because N is small, that ignores how probability works.
>Why not use use N >= 100 or not at all.
Because N > 100 is a totally arbitrary value. The size of N that you need is totally dependent on the effect size, not an arbitrary value. You'd miss lots of real but small effects by having an N of only 100. You'd miss out on lots of real but large effects which can be observed at N < 100.
Basically this approach would severely limit the branches of research we can do, for a questionable gain in statistical reliability.
This seems to be such a case. I wonder whether it was set from the beginning with more funding in mind.
Say you're a tribal chieftain. One of your men (one among thousands) finds a berry bush. Another eats a berry from the bush. He promptly dies. You're a scientific tribe, though, and you know N must be high before you conclude the berries are poisonous. So, one of the other guys eats another berry. He, too, dies immediately. The process repeats until it's you considering the bush. How does the situation change if the number of people before you was 19 vs 21?
I agree with nabla's comment that the real problem is our inability as laypeople to properly understand (or even properly upvote) scientific work.
What's unfortunate is that the research gets published at this stage beyond the scientific community that understands how inconclusive it is. Because in the long run that just contributes to the erosion of public trust in the scientific method.
These are preliminary studies: someone says, "Let me try something," it appears to work, and they report what they saw, with the resources they had.
As a result, someone organizes follow-on studies with more resources, which costs much more. At that point, the effect usually disappears, due to the greater power of the more careful study.