I am eager to read this piece!
Even though I am afraid to see it confirm that btrfs still struggles to catch up…
The 2016 bcachefs benchmarks[0] are a mixed bag.
[0]: https://evilpiepirate.org/~kent/benchmark-full-results-2016-...
Facebook using it doesn't mean anything since they are probably using it for distributed applications. Meaning the entire box (including BTRFS) can just die and the cluster won't be impacted. I really can't imagine they are using BTRFS on every node in their cluster.
Just because Facebook is fault tolerant doesn't mean we don't care about failures. We actively run down any issues we hit, so while it doesn't have much of impact in the short term, we don't just ignore issues.
And also Red Hat hasn't contributed much to btrfs in years, and Oracle has one developer working on it. We are constantly working on it, and now that my priorities have shifted back to btrfs I hope that we will start to close out some of these long term projects.
Just because some code is added to the main kernel, assuming it is, doesn't mean it will be propagated to the main LTS distributions anytime soon.
I don't think that follows for lots of reasons:
- If enough of your boxes die that you lose quorum (whether from filesystem instability or from unrelated causes like hardware glitches), your cluster is impacted. So, at the least, if you expect your boxes to die at an abnormally high rate, you have to have an abnormally high number of them to maintain service.
- Filesystem instability is (I think) much less random than hardware glitches. If a workload causes your filesystem to crash on one machine, recovering and retrying it on the next machine will probably also make it crash. So you may not even be able to save your service by throwing more nodes at the problem. A bad filesystem will probably actually break your service.
- Crashes cause a performance impact, because you have to replay the request and you have fewer machines in the cluster until your crashed node reboots. It would take an extraordinarily fast filesystem to be a net performance win if it's even somewhat crashy.
- Most importantly, distributed systems generally only help you if you get clean crashes, as in power failure, network disconnects, etc. If you have silent data corruption, or some amount of data corruption leading up to a crash later, or a filesystem that can't fsck properly, your average distributed system is going to deal very poorly. See Ganesan et al., "Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to Single Errors and Corruptions", https://www.usenix.org/system/files/conference/fast17/fast17...
So it's very doubtful that Facebook has decided that it's okay that btrfs is crashy because they're running it in distributed systems only.
"Mason: The easiest way to describe the infrastructure at Facebook is that it's pretty much all Linux. The places we're targeting for Btrfs are really management tasks around distributing the operating system, distributing updates quickly using the snapshotting features of Btrfs, using the checksumming features of Btrfs and so on.
We also have a number of machines running Gluster, using both XFS and Btrfs. The target there is primary data storage. One of the reasons why they like Btrfs for the Gluster use case is because the data CRCs (cyclic redundancy checks) and the metadata CRCs give us the ability to detect problems in the hardware such as silent data corruption in the hardware. We have actually found a few major hardware bugs with Btrfs so it’s been very beneficial to Btrfs."
The sentence: "We also have a number of machines running Gluster, using both XFS and Btrfs." seems to imply Facebook is not using it heavily for actual data storage. What I distill from this (which is obviously my personal interpretation) is that Facebook mostly uses it for the OS and not for actual precious data.
See also https://code.facebook.com/posts/938078729581886/improving-th...
"We have been working toward deploying Btrfs slowly throughout the fleet, and we have been using large gluster storage clusters to help stabilize Btrfs. The gluster workloads are extremely demanding, and this half we gained a lot more confidence running Btrfs in production. More than 50 changes went into the stabilization effort, and Btrfs was able to protect production data from hardware bugs other filesystems would have missed."
But I think the inference to make is that Facebook trusts btrfs to increase reliability, not that Facebook trusts their distributed systems to cover for btrfs decreasing reliability to gain performance (or features).