Hacker News Reader

Comment by iskander | Hacker News Reader

Science often works with data set sizes that make the standard ML approach unworkable (e.g. 12 measurements which took months and thousands of dollars to make)

View on HN

The solution isn't proposed to use ML. It's simply an analogy.

If you're performing statistical inference on sample sizes of 12 you can't then be surprised by a lack of predictability.

And I'm pretty confident the majority of experiments are using sample sizes larger than 12.

I don't think iskander though you were advocating using ML, but generally saying that the approaches _usually_ used in ML would not work in some datasets which are way too small.

A 12-point dataset is not desirable _per se_, but sometimes it's all one has - as is the case, for example in health-related research.

In which case, you can't really afford train and test splits but you can look at approaches like leave-one-out to estimate what your sampling error looks like.

And, more generally, bootstrapping, which is a class of methods for re-sampling one's data in order to estimate uncertainty.

https://en.wikipedia.org/wiki/Bootstrapping_(statistics)

I think leave one out is actually a special case of k-fold cross validation where k is equal to the number of datapoints. But yeah, you can use bootstrapping, cross validation, the jackknife. There's a large toolset there, and I trust an analyst who explains what motivated them to choose a certain tool rather than one who just uses a tool because it's what they always use or it's expected of them.

>And I'm pretty confident the majority of experiments are using sample sizes larger than 12.

It definitely varies by field, testing 3 doses and a negative control with 3x replication would be a pretty good experiment for testing a drug in mice. It's plenty of data for identifying a drug that actually works, as long as your statistical tools aren't designed with expectation of thousands of samples.

Reply on news.ycombinator.com