Hacker News Reader

Comment by BoiledCabbage | Hacker News Reader

The solution isn't proposed to use ML. It's simply an analogy.

If you're performing statistical inference on sample sizes of 12 you can't then be surprised by a lack of predictability.

And I'm pretty confident the majority of experiments are using sample sizes larger than 12.

View on HN

>And I'm pretty confident the majority of experiments are using sample sizes larger than 12.

It definitely varies by field, testing 3 doses and a negative control with 3x replication would be a pretty good experiment for testing a drug in mice. It's plenty of data for identifying a drug that actually works, as long as your statistical tools aren't designed with expectation of thousands of samples.

I don't think iskander though you were advocating using ML, but generally saying that the approaches _usually_ used in ML would not work in some datasets which are way too small.

A 12-point dataset is not desirable _per se_, but sometimes it's all one has - as is the case, for example in health-related research.

In which case, you can't really afford train and test splits but you can look at approaches like leave-one-out to estimate what your sampling error looks like.

And, more generally, bootstrapping, which is a class of methods for re-sampling one's data in order to estimate uncertainty.

https://en.wikipedia.org/wiki/Bootstrapping_(statistics)

I think leave one out is actually a special case of k-fold cross validation where k is equal to the number of datapoints. But yeah, you can use bootstrapping, cross validation, the jackknife. There's a large toolset there, and I trust an analyst who explains what motivated them to choose a certain tool rather than one who just uses a tool because it's what they always use or it's expected of them.

Reply on news.ycombinator.com