Yes, which is why using other metrics to measure performance would be better (specially offline).

Our startup is also currently building a product which will help publishers improve ad performance by using continuous testing, machine learning and optimization.