What were some of the other hurdles you faced in your Ceph deployment?
What were some of the other hurdles you faced in your Ceph deployment?
I found Ceph's error messages very hard to debug. Just google around a bit for the manuals of how to deal with buggy or fully defective drives. There's a lot of SSH'ing in, running vague commands looking up id's of drives and matching them to linux device mount points and reading vague error logs.
To me as a high level operator it feels it should be simple. If a drive supplies a block of data, and that data fails its checksum, it's gone. The drive already does its very best internally to cope with physical issues, if the drive couldn't come up with valid data, it's toast or as close to toast as anyone should be comfortable with. So it's simple, fail a checksum, out of the cluster, send me an e-mail, I don't get why Ceph has to be so much more complicated than that.
So RBD is easy, S3 is somewhat more complicated as you need to run multiple gateways, but still very doable. The FS stuff also needs extra daemons, but I have not yet tested it.