Hacker News Reader

Comment by solveit | Hacker News Reader

You're absolutely right, but the effect size is pretty damn impressive and I expect this to at least partially replicate.

View on HN

Having a small sample and a large effect doesn't mean you're guaranteed at least some effect on a large sample. If anything, a very large effect is more likely to indicate a flaw in your method.

Yes, but effect sizes are relatively harder to hack/screw up (knowingly or otherwise) than p-values. My totally anecdotal impression is that findings showing large effect sizes seem to replicate better than findings showing highly significant effects.

It could be an example of publication bias, from the wiki article on effect size [1]:

An example of this is publication bias, which occurs when scientists report results only when the estimated effect sizes are large or are statistically significant. As a result, if many researchers carry out studies with low statistical power, the reported effect sizes will tend to be larger than the true (population) effects, if any.

[1] https://en.wikipedia.org/wiki/Effect_size

Giant effect sizes are easy to hack in biological sciences. All you have to do is conduct an in vitro study.

Like they did in this paper.

Excessively large effect sizes are a warning sign in and of themselves. A good article on this is here: [0] -- simply if the effect is too huge, it becomes incumbent on the researcher to explain why no-one has noticed this before. Methane is a gas with a distinct and noticeable smell. Seaweed is plentiful in many places cows are kept. If the effect was really as dramatic as claimed you'd expect that it'd be well known that letting cows eat seaweed made them less smelly, and the authors of the original study make no attempt to explain why this might be the case.

0: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...

> "it becomes incumbent on the researcher to explain why no-one has noticed this before"

Farmer: huh, we'll I'll be damned. Who in the hell woulda thought to feed them seaweed."

If I'm not mistaken the original inspiration for this research was precisely that farmers noticed their cattle (or perhaps it was sheep) were happier when allowed to graze in fields which happened to be along the shore, allowing them to naturally eat quantities of sea weed.

> Methane is a gas with a distinct and noticeable smell

Methane is odorless https://en.wikipedia.org/wiki/Methane

Interesting, thank you! I was surprised by this, as it's the major component of natural gas (as used widely for cooking here), and this has a distinct smell. I went looking however, and discovered that's actually because Ethyl Mercaptan is added specifically to make it detectable by humans. I never realised this, and does indeed explain why it's plausible no-one would have noticed such a large effect.

We add mercaptan (an horrible smelling, innocuous gas) to natural gas (methane), so we get a chance to notice leaks before an explosion.

Maybe? But an effect which relies on a link between diet and farting isn’t exactly remarkable or even surprising.

My dog farts the most noxious gas if he eats apple or chicken.

In a small sample, the observed effect needs to be large in order to cross the threshold for statistical significance. It could still be a fluke and the true effect zero or even negative.

What's the tendency when a study with a small sample size that show a large observed effect get performed on larger samples? Has it been studied?

It depends. If the original study was published only because it had a positive result (significance filter, usually true), we expect the effect to be smaller in a larger replication. It's called Type M error, with M for magnitude. And, yes, large scale replications typically show smaller effects or no effects. See this paper for details: https://journals.sagepub.com/doi/full/10.1177/17456916145516...

Reply on news.ycombinator.com