Using everything modern and useful!

Performance testing has always had problems with isolation, mocking, covariance of services etc. I’ve just spent two weeks driving down latency across our framework, and autoresearch was definitely not a viable path. Most of these loops have these logarithmic, non-step change curves.

Just like with actual research, the real benefits came from me asking why is it like X and not like Y?

The original performance testing framework Claude created to bench our different versions against did not even mock high variance provider calls…