Hacker News Reader

Comment by kbutler | Hacker News Reader

This is the standard recommendation, but does it really help?

You do that test on the second subset, and then you discard every theory that doesn't pass both the 1600 set and the 400 set.

So you end up with the predictions that pass all the data in your original 2000 samples.

Is it really any better at generalizing to new data? If so, can you just evaluate your theories on the unpartitioned data by doing randomized subset testing after-the-fact?

View on HN

It depends on how often you go to the well. This is why things like Kaggle have limitations on how often you can submit entries for evaluation. This is why some studies go with a three-way split of train-calibration-test. And you can look at things like cross validation, boot-strapping etc. Nothing is perfect. As soon as you lock yourself into a one-size-fits-all way of validating your out-of-sample validity, you'll find lots of people who are optimizing for whatever test you come up with, not out-of-sample validity. It pays to pay attention to how well the researcher understands and deals with their particular set of problems, not how well they meet some arbitrary criteria. (Which is exactly how we ended up having all these problems with p-values! P-values aren't inherently bad, they're just applied indiscriminately because It's The Way We Do Things. If we start indiscriminately applying test/train splits we'll end up at a different place with its own set of problems. The chief problem is the lack of discrimination, not the specific tools under consideration.)

Probably would be helpful to see the math on it to confirm, but I believe if it's a spurious relationship, the probability it will pass both the 1600 & the 400 are less than the prob of it passing the 2000.

It may (or may not) be less, but if scientists just discard the tests which don't pass both, then passing both becomes the new prior, and we go back to the same situation.

Reply on news.ycombinator.com