Hacker News Reader

Comment by Vinnl | Hacker News Reader

So basically, it means waiting a while before performing the test again? Since "larger sample size" mostly just means "keep the test running for longer", so the difference is that in that case, the "second" test is very close in time to the "first" test?

View on HN

Not asking me, but yes. I think the underlying issue is whether you assume the process that generated the data yesterday is similar enough to the process that generated data today. In truth, all data points are unique and all data is high dimensional data. It only becomes workable in low dimensions once we make assumptions like "the process that produced this point yesterday is the same as the process that produced this second days point today." Separating the tests makes it slightly less difficult to test for violations of these assumptions (e.g., model drift). But otherwise it seems silly and arbitrary to commit yourself you two medium tests instead of one big test. At least two medium tests let you filter out likely failures if you allow yourself to quit midway. (This whole business of tests and halting is silly anyhow.)

Right, thanks. And am I correct in thinking that your final sentence before the parentheses means that two medium tests are worse, because you don't want to filter out likely failures?

Not entirely. Even if run right after one another, running two tests buckets the results into early/late, which simply using a larger sample set does not.

Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.

So what would the conclusions be in the two-test and large-test case? My assumption would be that if one of the two tests would fail, then the correlation in a large test would also have been so low as to mean no effect?

The issue is that you probably wouldn't do two tests if you'd planned a short one. So you'd randomly get an overly strong, or overly weak, result.

The actual effect depends on your data and how noisy it is, and if that correlates to external factors, etc.

You need to know the characteristics of your data then you can decide on your desired confidence and only then pick a test designed to give you that based on your data.

Reply on news.ycombinator.com