This is a fascinating read. We run a Ceph storage cluster for persisting Docker layer cache [0]. We went from using EBS to Ceph and saw a massive difference in throughput. Went from a write throughput of 146 MB/s and 3,000 IOPS to 900 MB/s and 30,000 IOPS.
The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something.
It’s been a massive improvement for our caching system.