Hacker News Reader

Comment by Shenglong | Hacker News Reader

I'm not all that familiar with social tests like this. Is n=120 generally considered to be a large enough sample size to make any claims at all?

View on HN

It always depends on the effect size.

Another important question is always, 'if this effect is true does this generalize to the overall population'. While I haven't read the paper, I bet that the test population were composed almost completely of undergrad students. So much of social science experiments are based on testing done on student. And there isn't anything wrong with that per se, but we must always be careful when trying to infer results, and drawing conclusions on the greater population, or human kind as a whole.

External validity is an interesting but quite different question. There are any number of other interesting but distinct features of clinical trials that we could talk about.

For some things (socialization, substance use patterns, etc.), you'd expect different results on college students than on other populations, so you'd need to run additional experiments to determine if being a college undergrad moderates whatever effect you study.

For other things, like perception, drug effects, etc., it seems less likely that this is actually a problem, because there's no reason to believe the neural pathways that acetaminophen operates over are different between any two humans.

n=120 pales utterly beside the observation that it is totally unclear what they are actually manipulating or measuring. The design is completely hapless.

If you really need to detect effects that are so small they are meaningless for almost any practical purpose, then you might need thousands of people in the study.

Just because you think a study seems too small doesn't mean it is scientifically invalid. There are actually statistical tests you can use just for the specific purpose of determining validity. These are much better than some intuitive hunch you might have about the study.

Xcelerate is correct. In most studies of any size (in my experience) you start by guessing the effect size that you think you might see. You then try to design your study with a certain power to detect an effect of that size.

You can try to manipulate your power in a variety of ways through the sample selection process (which can ultimately modify your external validity). The major variable that most people manipulate is simply the size of the sample (i.e., the number of people). You need a sample of a large enough size such that you could even hope to detect the effect you expect to see.

Ultimately, one can say, "This study had 80% power to detect an effect size of 30%."

Reply on news.ycombinator.com