The Disadvantages of A/B Testing
blog.zuupy.com
blog.zuupy.com
"2. A/B testing measures only quantity, not quality." If you have anything that can quantify your 'quality' of customers then include it in your metrics (I understand this can be hard). If you don't then it is not quantifiable and any analysis of 'quality' won't be based on statistics.
I agree on the other points. Overall I think A/B & multivariate testing is a good approach but can significantly be improved.
It seems like we use it in 'one size chunks' rather than as continuous improvement. Yes we do it ever 'x' days/weeks. What would be interesting is some kind of machine learning algorithm applied to A/B testing. It has the success metrics, you set a rotation period and it focuses in on the part that works.
You would have to integrate a factor for #5 "Things change" so that the algorithm has the flexibility to adjust to those changes.