File systems unfit as distributed storage back ends: lessons from Ceph evolution

I think a larger problem with new architectures is that consumer adoption follows price/performance/power, not any inherent architectural quality. The architecture we're stuck with is the one that most hardware devices get sold with (x86/ARM right now). That gets determined by hardware OEMs, which in turn make their decisions based on what'll help them sell the fastest devices with the lowest power consumption for the least money. So something like RISC V is fascinating and quite elegant, but until there are RISC V chips that are cheaper and faster than Intel ones, it remains an academic curiosity. Then if you're a compiler writer, you gotta work with what you've got for an installed base, and you can't really get adoption for a new language unless it lets startup founders unlock new markets because your combination of development velocity + execution speed lets them do things they wouldn't otherwise be able to.

1.) automatically defining SoA/AoS transformations for data types in the language, and

2.) automatic profiling of code execution counts & array lengths to determine when inserting such a transformation would save more time than the transformation costs.

3.) native SIMD primitive types

4.) GPU backend a la Futhark

5.) native syntax for parallel execution, where the compiler enforces no data sharing (or read-only shared data) between executions.

6.) [more speculative] pluggable backend for this syntax so that you could run these concurrent executions sequentially, using SIMD instructions on the same thread, in multiple threads, on the GPU, or spread across multiple machines in a cluster without altering the code. That'd let you scale up from a few K to a few T without altering your code.

7.) I/O mechanisms that default to mmap, 0-copy vectored I/O, or RDMA as appropriate.

8.) An unfold primitive in either the language or standard library that lets you compute multiple values with a single traversal over a list, ideally while still keeping each individual computation isolated and modular. I find this is a really common pattern in my Kotlin code, and the biggest culprit for still writing manual accumulator loops.

9.) Support for multi-pass initialization in data structures, where you can hold some fields as uninitialized for a later computation, but mark the whole structure as immutable once every field has been computed.

10.) A standard library that uses these parallelization techniques under the hood so you get really good speed for common operations but it just works. For example, Boyer-Moore can be done on the GPU. Many common regexps can be reduced to string search on a common left prefix + a DFA on potential matches, where the prefix search can be done with Boyer-Moore. That opens the possibility of blazingly fast search over gigabytes of data for common regexps like /<a href=["']?([^"'\s])["']?>(.+?)<\/a>/. Many times the algorithms you'd use when your haystack is ~1G are very different from the algorithms you'd use when your haystack is ~1K, which goes back to the utility of an integrated profiler.

It occurs to me that a lot of these boil down to the inefficiency of doing lots of traversals of arrays in a world where touching memory is usually the critical path. So for example, this is slow but modular:

    val high = oneBillionFloats.max()
    val low = oneBillionFloats.min()
    val total = oneBillionFloats.sum()
    val average = oneBillionFloats.sum() / oneBillionFloats.size()

    var high = Float.MIN_VALUE
    var low = Float.MAX_VALUE
    var total = 0
    var count = 0
    oneBillionFloats.forEach {
      high = maxOf(high, it)
      low = minOf(low, it)
      total += it
      ++count
    }
    val average = total / count

*I was recently part of an effort to add FreeBSD testing to DPDK because SPDK's FreeBSD test agent kept failing when we updated DPDK. I'm a core maintainer for SPDK, which is basically DPDK for storage.

> If any of those filesystems were truly adequate there wouldn't have to be so many.

Diversity is not bad. Different filesystems can have different design points, e.g. read-heavy vs. write-heavy workloads, optimized for different kinds of media, realtime requirements (traditionally an XFS strength) etc. That's a good thing. What's a problem is that there are too many filesystems trying to be all things to all users of a too-general API, and too many FS developers competing with each other on long-irrelevant microbenchmarks instead of making useful progress.

> Microsoft's failure

...is Microsoft's.

> Recently ... pwrite, pread

Those have been standardized since 1998, and I'm pretty sure I remember them existing on some systems before that. The fact that they're not recent is IMO the real problem. There have been attempts to create useful new abstractions such as the setstream API from CCFS[1], but they're too few and rarely gain traction.

Personally, I think local filesystems need to adapt in a bunch of ways to be a better substrate for higher-level applications - including but not limited to distributed filesystems. OTOH, I don't think putting complex transactions at that level is a good choice. There's enough innate complexity there already. Adding transactions there just increases inertia, and if filesystems offered better control over ordering/durability the transaction parts could be implemented in a layer above.

[1] https://www.usenix.org/system/files/conference/fast17/fast17...

First of all ZFS is not a “linux” filesystem. Second, available choice of filesystem is a strange thing to use as justification that previously existing filesystems are “inadequate”. By what criteria are you establishing adequacy? Success of technologies is often not dependent solely upon their technical excellence but on a variety of factors that may have nothing to do with the technology itself. (Betamax vs VHS, etc.)

Why are there so many filesystems? Because everybody starts from scratch.

Then everybody writes code first and later a description of the filesystem.

We need a way to learn from previous mistakes and then find a way to fix those problem.

While I get your point, I would like to point out that ZFS was developed by Sun (now Oracle). I've used ZFS for years from a data-integrity and array-mirror perspective and love it. No other file-system you mentioned next to it gives me the confidence that ZFS does (maturity, stability, etc).

I'm not sure about RocksDB (I think the "levels" compaction runs in the background by you can control it).

[1] https://www.sqlite.org/malloc.html#_setting_memory_usage_lim...