Good point - user experience may include cached and non-cached I/O so it would be beneficial to include both in this type of analysis.

The benchmarks binaries, configurations and runtime settings were generally consistent for instance types of the same size across services, but we didn't verify efficacy of the benchmarks as they ran.