This is awesome. I was looking at benchmarking speed and quality myself but didn't go this far!
I wonder about Claude Instant and Phi 2?
Modal.com for inference felt crazy fast, but I didn't note the metrics.
Good ones to add?
Replicate.com too maybe?