I assume for three reasons:

  - small file performance (if they're not storing the git objects as packed)

  - POSIX compliance

  - Their on-prem installations for customers

For posix compliance, you can use s3fuse/gcsfuse or put Avere in front, but you're either paying more for it or you're not getting the best performance.

For small file performance though, no object storage system is particularly tuned for that (often the granularity is more like MBs, and at least with GCS you're talking ~100ms per write).

Finally, they need to be testing and supporting customers who want to run the whole thing "at home" wherever that is. So they would still need to run Ceph!

Disclosure: I work on Google Cloud.