It's impossible to reliably satisfy anything in a distributed networked anything. There aren't costs as much as tradeoffs. Reliable, fast, accurate: pick one.
But formulated in that language, my comment becomes: POSIX picks less than useful trade-offs for distributed file systems.
An extreme example: POSIX's atime gives you something you seldom need, but imposes a huge burden. That's such a bad trade-off, that approximately no-one implements the original spec, even if nothing is distributed at all.
I had it in my head that modern flash actually does read in a way that involves writing, and we just paper over that for the sake of the abstractions we use. Am I wildly misremembering?
Sometimes blocks may have to be read and re-written elsewhere to make room... and this is what makes using TRIM and having cache on-disk so important.
He basically stated the CAP theorem. It's been proven mathematically.
Essentially if you distribute your data, you have to choose some degree of consistency versus availability.
I'm not fully familiar with POSIX requirements for filesystems, but having atomic changes and good performance and a host of other aspects to keep the illusion of available and consistent files in a distributed store runs right through the CAP theorem.
As an example of how to do better, squint a bit and have a look at git interpreted as (part of) a distributed data store.
Git doesn't pretend that you can make a change in one location, and have it atomically show up in all locations. On the other hand, Git does allow you to make changes to multiple files, and only have them show up together or not at all.
Git also has explicit mechanisms for resolving conflicts when multiple people tried to make changes to the same files.
Git doesn't get around the CAP theorem; it just has much better trade-offs than POSIX, because Git was designed explicitly for distributed use.
POSIX also doesn't allow you to have different trade-offs for read-only and read-write data.
In any case, I have seen enough of these "A, B, C : pick N" (where 0 < N < 3) mantras shot down (usually by re-examining whether the constraints/assumptions of the original mantra apply in a specific situation, or just through plain algorithmic engineering) that whenever someone drops something like this, I mostly tune it out.
In the case of distributed network <anything>, implementing programs in languages that help you manage important invariants automatically (and preferably not at run-time, but at compile-time) is key in creating robust systems that are both reliable, and fast.
But if you have such tools, and you are trying to also meet a set of standards that were designed without any care for distributed computing, then...yeah.
Reliable: A system can be available but not reliable; if a machine is down 6 minutes every hour, it's 90% availabile but less than 1 hour reliable. A networked distributed system can be reliable but not accurate if you keep it online while it slowly accrues errors, such as the mentioned link/unlink race condition. The machine still works, it just has occasional corruption. It tends not to be very fast if it's reliable.
Fast: You can do whatever you want quickly. Think UDP, or filesystems that don't commit their log journal frequently. Tends not to be reliable or accurate.
Accurate: Whatever you do has no corrupted state. Think ACID. Most systems don't do this because it's expensive/complicated.
> Think ACID. Most systems don't do this because it's expensive/complicated.
Is this true? A lot of systems are built on databases.
> Accurate: Whatever you do has no corrupted state
How does accuracy work if the system is distributed and there's a partition, and two node get different data? Presumably it's reliable if they're different, as accuracy is a different item?
> Fast: You can do whatever you want quickly. Think UDP, or filesystems that don't commit their log journal frequently. Tends not to be reliable or accurate.
Why wouldn't this be reliable?
> How does accuracy work if the system is distributed and there's a partition, and two node get different data?
You don't allow writes during a partition. All nodes (or a quorum) must confirm on each write, and if that fails then a partition is detected. Everything that was accidentally written before you detected partition is thrown away. Usually you just have one writer node for simplicity.
> Fast: > Why wouldn't this be reliable?
If you prioritize reliability you will need a slower method to ensure reliability. TCP is slower than UDP so it can be reliable.
Wouldn't this be done better with a majority/minority approach, where if enough nodes are still communicating then they still work?
> If you prioritize reliability you will need a slower method to ensure reliability. TCP is slower than UDP so it can be reliable.
Agree - that is true in the TCP sense of the word reliable. But in your sense of the word, as I understood it, the TCP/UDP difference is more like "accuracy", is it not? Reliability is more like availability, as I read it in your previous comment.