The benchmarking repository has config files used for the test, and they did not use the tile cache feature.
Regardless, I don’t think caching came into play here, at least according to how I’m reading this repository. I would expect cold caches for everything.