Great point. In addition to caching, there's potential for branch prediction effects.
This is why the JMH samples warn against including loops[0] of the operations you're trying to measure, and have an API that's designed to avoid the programmer having to every write their own looping code.
[0] https://hg.openjdk.java.net/code-tools/jmh/file/7bc6011260aa...