"k is the expected number of future users who will be exposed to a result"
Does this mean that this approach does not make much sense if your estimate of k is totally wrong?
How do you estimate k?
Does this mean that this approach does not make much sense if your estimate of k is totally wrong?
How do you estimate k?
One is to estimate it based on the number of daily visitors your site gets, and then estimate how long you will run the winning alternative in the campaign.
He also proposes: 'perhaps k should be assesed, not as a constante, but as an increasing function of |y|/n, since the more striking the treatment difference indicated the more likely it is that the experiment will be noticed... One way of introducing such a dependence of k on |y|/n is to assess k+2n as a constant.'
This actually simplifies the math somewhat, and you can see the full details in Anscombe's paper cited in the blog.
If k is an estimate of how much traffic I will ever see, it seems like I'm going to be calculating the Phi-inverse of approximately 0.
Where can I see Anscombe's paper online? (It was published in 1963; it's not linked in the blog post, just cited.)
In practice, we don't actually ever run the winning idea for ever. We do website re-designs periodically, we test new ideas, business needs change. So we can pick a reasonable value for k based on these constraints.
Alternatively, you can get better performance by _not_ picking a stopping criteria, and dynamically choosing which homepage to show. As soon as one idea appears to be doing better, you start showing that to more users. By choosing the appropriate adaptive sampling strategy, you can reduce regret to be less than if you have a constant sampling strategy. However, for many people the adaptive strategy may be more trouble to implement than it is worth.
The most important takeaway is to _not_ use repeated significance tests to determine experiment termination time. Either use the Anscombe bound with an appropriate k, or fix the sample size before starting the experiment.