Nice article. One question though:
> Perform a second ‘validation’ test repeating your original test to check that the effect is real
Isn't this just the same as taking a larger sample size?
> Perform a second ‘validation’ test repeating your original test to check that the effect is real
Isn't this just the same as taking a larger sample size?
The difference is time, so the populations involved are 'the people that used the site last month' and 'the people that used the site last week'. Usually your assumption is that these are comparable, but that's not necessarily true. Furthermore, the effect is only meaningful if that assumption is true, for most business cases (i.e. you want this effect to hold in the future).
In practice, a lot of effects do disappear when you repeat a test because there was some unaccounted for factor that varied between them. It's a good sanity check.
But you're right, much of the purpose is to discover mean regression. This is something that happens more often than you'd expect because you tend to be focusing large effects, many of which will simply be due to chance.
Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.
The actual effect depends on your data and how noisy it is, and if that correlates to external factors, etc.
You need to know the characteristics of your data then you can decide on your desired confidence and only then pick a test designed to give you that based on your data.