https://speakerdeck.com/takutakahashi/ceph-on-zfs
I was impressed when I found this slide deck and it was in japanese, and from a thoroughly japanese company here. If you don't happen to know japanese, just go like... 3 slides in and you can see the creator is from GMO, also there are some images.
If you're not into infra you might as well stop reading here, but basically the idea is to get durable crash-resistant replicated storage on kubernetes by actually combining two CSI storage providers:
- Run ZFS as your on-disk file system (for absolute durability without bitrot and some of the other benefits) for each node
- Add the openebs/zfs-localpv[0] to dynamically provision zvols with say xfs or ext4 to make drives for ceph to consume
- Add Ceph via Rook[1] to get synchronous remote replication on every write (if you want it), and give ceph the drives you made (the zvol(s))
- Expose storage to your kubernetes cluster that is replicated (depending on how you set up ceph) and easy to back up and/or duplicate at the node level (w/ zfs send). You can back up at multiple levels, nodes themselves (with some replication lag since zfs doesn't so sync replication) and the workloads that run on them (which are sync replicated across the network).
There is some overlap in these tools -- ceph's relatively new bluestore engine does checksumming now and self-heals, so there's not so much reason to use ZFS at the lowest level at all, but ZFS does have some features that ceph doesn't, and is somewhat easier to manage with less moving parts in user space.