I've performed, and also debugged, a lot of these cloud comparison benchmarks, and it's very, very easy to have bogus results due to some misconfiguration.
What is the total file size for the fio runs? (And, what is the intended working set size?) Was fio configured to bypass the file system cache and perform I/O directly to disk? (And if so, what is the rationale for bypassing it?[1]) Was iostat or other tools run during the benchmark to confirm that fio was configured and operating correctly, and that the results could be trusted? Was the same version of fio used, and built with the same compiler (same binary?).
The 118 page report does not include the actual fio commands used.
[1] Disk I/O latency can be a serious issue in cloud environments, and one that vendors can address by incorporating additional levels of storage I/O cache (eg, in the hypervisor). Picking benchmarks that bypass discourages vendors from doing this, which is not ultimately good for our industry.
Excellent work, thanks for doing a quality job on this.
I know what I'm suggesting is not the current norm for cloud evaluations. And I believe the current norm is wrong.
The more important question is how the benchmarks were analyzed -- what other tools were run to confirm that they measured what they were supposed to?
The benchmarks binaries, configurations and runtime settings were generally consistent for instance types of the same size across services, but we didn't verify efficacy of the benchmarks as they ran.