The capability of A/B testing should have made this experimentation a one-armed bandit problem, which is far more ammenable.