Hacker News Reader

Comment by solveit | Hacker News Reader

Yes, but effect sizes are relatively harder to hack/screw up (knowingly or otherwise) than p-values. My totally anecdotal impression is that findings showing large effect sizes seem to replicate better than findings showing highly significant effects.

View on HN

It could be an example of publication bias, from the wiki article on effect size [1]:

An example of this is publication bias, which occurs when scientists report results only when the estimated effect sizes are large or are statistically significant. As a result, if many researchers carry out studies with low statistical power, the reported effect sizes will tend to be larger than the true (population) effects, if any.

[1] https://en.wikipedia.org/wiki/Effect_size

Giant effect sizes are easy to hack in biological sciences. All you have to do is conduct an in vitro study.

Like they did in this paper.

Excessively large effect sizes are a warning sign in and of themselves. A good article on this is here: [0] -- simply if the effect is too huge, it becomes incumbent on the researcher to explain why no-one has noticed this before. Methane is a gas with a distinct and noticeable smell. Seaweed is plentiful in many places cows are kept. If the effect was really as dramatic as claimed you'd expect that it'd be well known that letting cows eat seaweed made them less smelly, and the authors of the original study make no attempt to explain why this might be the case.

0: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...

> "it becomes incumbent on the researcher to explain why no-one has noticed this before"

Farmer: huh, we'll I'll be damned. Who in the hell woulda thought to feed them seaweed."

If I'm not mistaken the original inspiration for this research was precisely that farmers noticed their cattle (or perhaps it was sheep) were happier when allowed to graze in fields which happened to be along the shore, allowing them to naturally eat quantities of sea weed.

> Methane is a gas with a distinct and noticeable smell

Methane is odorless https://en.wikipedia.org/wiki/Methane

Interesting, thank you! I was surprised by this, as it's the major component of natural gas (as used widely for cooking here), and this has a distinct smell. I went looking however, and discovered that's actually because Ethyl Mercaptan is added specifically to make it detectable by humans. I never realised this, and does indeed explain why it's plausible no-one would have noticed such a large effect.

We add mercaptan (an horrible smelling, innocuous gas) to natural gas (methane), so we get a chance to notice leaks before an explosion.

Reply on news.ycombinator.com