Hacker News Reader

> Because fsync() is so coarse

You can do some work in advance with sync_file_range. fsync is still needed to write out the metadata, but it'll have to do less work and you may be able to get better pacing and backpressure that way.

> Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them.

Many existing filesystems simply cannot support such an API because they're not journaling or CoW. Another issue might be that multiple transactions could overlap and be of unbounded size, so now you would need MVCC, exclusive locking or the ability for the kernel to provide feedback to userspace that it will immediately rollback the transaction due to resource limits or conflicts. This suddenly looks more and more like a database than a posix filesystem.

There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead. Lazy writes that the kernel can flush to disk whenever it feels like provide a lot of the performance that we take for granted when working with large amounts of data.

> And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing.

I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace.

> Many existing filesystems simply cannot support such an API because they're not journaling or CoW.

Thats fine. Legacy filesystems can just use a best-effort approach like they do now with 95% of programs. I already wouldn't put data I care about on ext2. Programs could detect transaction support with a syscall then decide whether to fallback to old behaviour, lose reliability or for a database maybe just refuse to run entirely.

> This suddenly looks more and more like a database than a posix filesystem.

POSIX filesystem apis aren't fit for use when you care about correctness. And as I understand it ZFS, btrfs and APFS already look more like databases internally than traditional posix filesystems. If that ship is sailing anyway we may as well get all the benefits we can.

> There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead.

Are there? I can't think of many. Maybe copying big files? We'll need to keep supporting the current unsafe-by-default posix API forever for current software anyway. I just don't think it should be the default for 95% of applications. When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes.

If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. It seems like a pity though, and I think the filesystem is worth salvaging.

And with the API I'm suggesting, I don't think large operations need to get much slower. The transactional API could just optimistically write at full speed to unlinked sectors on disk, then atomically update pointers to reference the new sectors when the transaction is committed. That wouldn't add much amortized overhead. I'm sure there's more complexity than that in practice, but it sounds pretty tractable and fast in the common case of a single writer. (And in the slow case where multiple programs try to fight over the same file, slow would be an improvement over the current behaviour of "probably everything breaks".)

> I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace.

Its still almost impossible to write a database correctly on top of the POSIX APIs. Especially if you want it to be correct. That seems fixable. We should fix it.

> If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree.

I don't mean everyone should use sqlite. We just need more safe abstractions over posix APIs that cover different use-cases. If you need to concurrently query and modify many small records, then sure, go with sqlite.

> Are there? I can't think of many. Maybe copying big files?

It's not just the big files. Millions of small files would cause some overhead if you actually tried commiting them instead of relying on best-effort behavior (see git case below). Temp and cache directories. Archival. Compiler artifacts. Often you don't even want them to actually hit the disk at all if you have enough RAM. A file can act as nothing more than named swap space.

> When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes.

Sure, that's for the canonical git storage, which rarely touches many files anyway. But if you checkout a worktree (e.g. by switching branches) then putting all the thousands of files from the .git object storage into the work tree does not need durability. You want to be able to work on those files as soon as possible, even if they're just in the page cache and not written to disk yet. The command shouldn't block on fsync.

> We just need more safe abstractions over posix APIs that cover different use-cases.

I reach for lmdb for a lot of small projects. It feels like failure though. We shouldn't need things like that just to deterministically save data to disk.

> The command shouldn't block on fsync.

Sure - some operations can and should discard durability for higher performance. Sounds great to me. But maybe you should opt in to that. It certainly shouldn't be the only behaviour the kernel provides. If Inkscape crashes halfway through saving my work, it shouldn't by default end up in a half-saved state where my file is permanently corrupted.

But we can have our cake and eat it too here. It can be as simple as, if you don't wrap your write() calls in a filesystem transaction, the behaviour stays the same as it always was. Or maybe the default behaviour is that a write call entirely happens or entirely doesn't happen, but you can optionally ioctl(NO_DURABILITY) or something for extra performance in git checkout.

I hear you that git checkout performance (and things like it) are important to you. Keeping the current performance of those operations sounds good to me too!

> CoW

Are you talking about copy-on-write or redirect-on-write, which seems to be the real copy on write ?

https://en.wikipedia.org/wiki/Copy-on-write

From the same article:

> Despite their names, copy-on-write usually refers to the first technique.

Yes, hence my question.

CoW filesystem is a standing term used to refer to things such as zfs or btrfs. They use the first kind.

>> Many existing filesystems simply cannot support such an API because they're not journaling or CoW.

So let's get rid of them. And that leads to "but what about all the software that hasn't updated to the new APIs?" Guess what?

Interestingly enough, that feature already exists and is being removed from Windows. From the transactional NTFS Wikipedia page[0]:

> Microsoft has deprecated TxF and stated that it may be removed in a future version of Windows

[0] https://en.wikipedia.org/wiki/Transactional_NTFS

Cool! Does anyone know why its being removed? The article says it supports atomic operations spanning multiple computers. Thats pretty tricky to implement correctly. My guess would be that its being removed because the feature is overbaked and it impacts too many parts of windows. But I'd love to know for sure if anyone has insider knowledge into the feature.

Basically nobody is using it:

> While TxF is a powerful set of APIs, there has been extremely limited developer interest in this API platform since Windows Vista primarily due to its complexity and various nuances which developers need to consider as part of application development.

> As a result, Microsoft is considering deprecating TxF APIs in a future version of Windows to focus development and maintenance efforts on other features and APIs which have more value to a larger majority of customers.

Actually, it is hard to remove it, since Windows Update is dependent on it.

"TxF is presently deprecated, but the functionality is still available. We don't recommend disabling this feature on the C: volume."

Microsoft supports APIs which were declared deprecated for decades.

However, I totally agree in "Basically nobody is using it" part. It's because it was not properly implemented in .Net Framework. And now, when .Net Core is multiplatform, i.e. worst of all worlds, chances it will be implemented are very close to zero. ReFS does not support TxF.

Windows has a lot of awesome APIs which are accessible only to C/C++ and are not advertised well.

This [0] mail from Linus might shed some light over this.

[0]: https://lkml.org/lkml/2009/3/25/632

Thanks. It looks like I have a deep philosophical disagreement with Linus:

> Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

I think he's wrong. Compiler / PL theory works. Distributed systems theory (TLA+, etc) allow us to write correct software. I've used fuzzers to find and fix bugs in databases and concurrency algorithms, and the result is robust, working code.

Theory can add complexity, but complexity added in one place can remove complexity from another. (Eg TCP is complex but reliable, in order delivery in the network layer makes the application layer much simpler and free of bugs.)

Linus's argument is that users will keep writing open(); write(); close();. I think he's half right - thats the right level of complexity the API should have, with no tricky, buggy fsync barriers. But we've known how to write simple, correct APIs for modifying data like this for decades. The answer is transactions. Keeping the kernel as it is just keeps us in a dark age of buggy software.

Wait, isn't TCP(/IP) the quick and dirty practical implementation that beat the much more theoretical OSI model ?

(And the same thing happened to GNU/Linux, so Linus would know something about it !)

Hah good question. When I was learning it in school, TCP/IP sure seemed complicated with frame numbers and window sizes and whatnot. It seems much more simple to me now. I suspect that everything sounds like "useless, complex theory" until you understand it and use it regularly, and then it mysteriously transforms into "obvious, simple practical knowledge".

Are transactional databases useless theory or simple and pragmatic? Hash tables? Nagle's algorithm? CRDTs? SSA compilers? Your answer probably says more about your experience than it does about the technologies themselves.

And thats why I don't trust Linus's rant. He's familiar with the garbage fire of the current posix filesystem APIs. I suspect deep down he's just arguing from status quo bias and doesn't like the "useless theory" of transactional filesystems because he hasn't seen the practical side of them and he hasn't thought through how they could be implemented efficiently in the kernel. And thats fine; but it might mean if we want transactional filesystems to happen in linux, it might make sense to start by implementing them in a BSD.

Funnily enough, I don't know that much about networking myself, I was mostly repeating what our networking teacher told us to illustrate the importance of practice vs theory.

Another Linus example would be git, which he allegedly threw together in a couple of weeks, compared to pijul, which seems to have required years of work, but now is going to have a very hard time to catch up...

> Wait, isn't TCP(/IP) the quick and dirty practical implementation that beat the much more theoretical OSI model?

Yes, it is. It even has its own (four-layer) model but the IETF People aren't nearly as interested in layering as the OSI people were:

Layering Considered Harmful

https://tools.ietf.org/html/rfc3439#section-3

Point being, this is why people who know history get a bit perturbed when people insist on shoving TCP/IP into the OSI Model, or, worse, insist that TCP/IP "implements" OSI somehow. OSI had its chance, it lost, and now we use POP and SMTP for email, not X.400.

The problem with that post is that it assumes that all application writers are writing crappy C.

Linus has tunnel vision about this, and it wasn't true in 2009 and is becoming increasingly untrue.

The people who need better tools to deal with this are the language implementers who almost always abstract over this. And since most language implementers target Windows and OS X as well as Linux, they are generally more aware of what better could look like than Linus.

> The people who need better tools to deal with this are the language implementers who almost always abstract over this.

Do they? What languages abstract away the filesystem or the OS's file abstraction?

Java? Python? Common Lisp? Others? I'm not dropping to direct system calls of each OS, so there is clearly some level of abstraction.

Barely. I don’t know about clisp but for the most part Java and python just provide thin shims over the posix syscalls. They don’t give you atomic transactional filesystem semantics.

Isnt the point of the article that these file systems are trying to implement something like this, with journals and write logs, and such as implementation details, but these have failure modes and bugs that mean these mechanisms can still give unexpected results. Its like saying, well software just shouldnt have bugs or be vulnerable to side effects.

Of course it is possible to mitigate these issues more than most file systems do, but at the cost of reduced space or performance efficiency. Thats what using a further abstraction like SQLite buys you, but its not for free. Thats why all the FS as database efforts in the early 2000s like Longhorn failed.

The point of the article is how if you want the filesystem API to write data safely, you need to jump through a lot of hoops. (Which, spoilers: most developers don't do. And even developers who try really hard often get it wrong because the APIs are so awful and poorly documented. Hence the ever present spectre of data corruption.)

According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar". This is the best scenario - without journaling, I can also end up with uninitialized junk.

The authors of the filesystem APIs consider this to be correct behaviour. But I can't think of any use case for this nonsense behaviour. Real data, in every file on your computer is structured. And structured files simply can't handle this kind of data corruption. Maybe this API has slightly higher throughput, but by the time you've peppered your code with fsync() calls to make it correct, you've killed performance. And working around that takes whole philosophies of threading and thousands of lines of code.

I want that write call to either complete completely or fail completely. Actually, I want more than that - for a bunch of successive write calls touching multiple files to collectively succeed together or fail together. Which is not what the article is saying.

> Its like saying, well software just shouldnt have bugs

The kernel shouldn't have bugs in its filesystem. We don't tolerate bugs in the scheduler, or in OpenSSL or in the CPU. If Postgres randomly corrupted its database every month or two, we would fix the bug. Or call it broken and use something else. If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel. And in the kernel, that code would be easier to optimize, and would be useful everywhere.

If it was a simple bug, we would fix it. The problem is that the bug is that we have the wrong abstraction. Why do we have the current filesystem API? The story I heard is it was written by some intern at Berkeley decades ago who thought it would be neat to have files and network streams use the same API. We know more now, and its past time to fix the mistake.

> Thats why all the FS as database efforts in the early 2000s like Longhorn failed.

I don't buy the cynicism. Just because microsoft overengineered longhorn (and successfully shipped TxF btw), doesn't mean the problem is unsolvable.

> According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar".

It makes a lot more sense if you consider it shared, concurrently accessed memory. If you try to access it without atomics you can get funny results. And if you kill one process in the middle you might be stuck with reordered accesses indefinitely. Something like ext4 provides more than that.

If you want ACID then you'll have to opt into that. The same as you have to do with concurrent programming (e.g. via std::atomic). But there are many cases where none of this is needed, where all you need is a bag to hold a few bytes for a while. E.g. cache files or large media blobs.

The only issue is ubiquity of the necessary tooling, or the lack thereof. Many standard libraries now ship with concurrency-safe data structures while you often have to reach for 3rd-party libraries for safe file abstractions if you don't want to code your own.

> If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel.

If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? Userspace needs many different levels of granularity. Sometimes integrity doesn't matter at all. Sometimes it only matters that writes make it to disk in correct order but it's ok for the last ones to get lost. And sometimes you need multiple processes memory-mapping a file and doing shared byte-level access and only random byte-ranges need to be committed to storage at any given time.

> It makes a lot more sense if you consider it shared, concurrently accessed memory.

But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously.

And even if you want to think about it like memory, we're missing all the tools we have for managing concurrently accessed memory like atomics and granular load and store barriers. And even then only about 1 in 10000 programmers know how to use those primitives to implement correct software anyway, so thats not exactly what I'd reach for when I think of good API design.

But even if you want to use that analogy, the filesystem API doesn't have APIs for granular memory barriers. Until recently all we had was fsync. Peppering your code with fsync calls destroys your performance even though it doesn't change your program's behaviour. And if you batch up your changes and then fsync returns an error, you have no idea which writes succeeded and which failed. inotify helps, but oh goodness does it get complicated.

> If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side?

This is a good question. I guess I don't really think much about caches, or processes sharing a pseudofile to communicate. For those programs, sure - the current behaviour is fine. I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk. Those programs shouldn't randomly end up in a corrupt state if power goes out at the wrong time. This is the core use case for the filesystem and I think its pretty reasonable for the kernel to make that case easy and correct by default, without resorting to storing all user data in every application in sqlite.

From Linus talking about this issue ( https://lkml.org/lkml/2009/3/25/632 ):

> The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like open(name, O_CREAT | O_TRUNC, 0666); write(); close();

Basically, the reality on the ground right now is that:

- Its almost impossible to reliably update a saved file right now without sometimes corrupting data. From the article, researchers found sqlite was the only correct database that existed based on their survey.

- Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used.

Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. A good API makes simple things simple and complex things possible. The posix API fails here. It makes the simple task of updating a save file almost impossible.

I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. But nobody seems to be making that argument, and thats not what reality looks like. So long as the filesystem API is advertised (and used) as the standard way to persist user data, it should do that in a simple, safe and correct way.

> But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously.

Well, but it is. Files can be opened multiple times and file descriptors can be shared between any number of processes and threads. mmap, atomics and msync also give you granular access.

Most people might not use it that way, but the thing is that it can be used that way. Just like RAM can be accessed randomly from many threads but higher-level languages limit what you're allowed to do concurrently. It's the same fast-but-unsafe-guts, use safe abstractions thing.

> I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk

Those 95% of the applications aren't necessarily those doing 95% of the write IO though. Looking at the activity on my system then most of the write traffic is a) logging (including messaging) b) firefox occasioally showing 1GB/s places.sqlite spikes (hah) c) temporary files d) docker e) p2p storage.

For logging O_APPEND's atomicity promise may be good enough. I don't think text editors need transactions either, they only need file-level atomic replace which can be handled by a single library function. Such functions should be available in more libraries, I can agree with that.

> Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct.

Why the kernel and not libc? Or some other userspace lib for that matter.

> I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data.

That might be a matter of perspective. open/write/close is perfectly fine if all you do is move 100s of RAW photos from your SD card to your file server. There's no rewrite-in-place happening, it's a one-time thing that at worst would leave you with one incompletely copied file if the PC crashes shortly after copying. And the SD card wasn't a bastion of reliability anyway. Docker layers and containers are quite discardable and I wouldn't even mind if they had 0 durability. git, sqlite and postgres already seem to be doing the right thing. Whether document processing software does, I don't know but I assume they put some effort into it with all their autosaving features. But, as mentioned above, all they really need is nicer standard libs not a new kernel API. That doesn't leave that many things I deem critical. Perhaps it is a little annoying if some of the configuration text files of various GUI applications get lost in a crash but on the other hand I wouldn't want them to fsync my preferences every time I tick a checkbox so I'd only want atomicity but not synchronous durability.

> - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used.

If people can't even be bothered to fsync then why would they be bothered to wrap IO into a transaction? After all APIs still would have to continue to work in non-transactional mode for performance and backwards-compatibility reasons so there would be no pressure on people to use transactions.

Maybe this also is a documentation problem since many developers don't seem to be aware of the non-durable-by-default behavior and different steps needed to get atomicity or durability. Man pages and standard libraries lack a high level overview of filesystem semantics. Some of it is noted in fsync() for example but not in rename()

Yeah if the current behaviour was well documented, and easy to use tooling existed in libc or something I’d be much happier. I think it could be optimized better in the kernel, but slow by default is better than wrong by default.

Perhaps io_uring is a way forward. Today you can already batch a bunch of IO calls and chain them together so that the next is only excuted when the previous one succeeds. With some additional hints journaling and similar filesystems could try putting such a batch into a separate transaction group. That would give you some crash all-or-nothing behavior but no rollback on error. It would also be on a best-effort basis, might only support some systemcalls etc. Then the situation could be improved piece by piece.

A I think reason is simple.

It works ok most of the time.

And if you really care about some data, you will stick it some database anyway.

And if you are going to stick data into database, you want to be able to select database that suits your needs best, and not be stuck with one provided by os.

B I think you are underestimating how hard is it to implement good and performant transactional database. I am not sure it belongs into kernel.

LMDB implements a persistent transactional API around a B+ tree in just a few thousand lines of code.

We put things in the kernel for several reasons - performance, but also consistency and interoperability. If every program embeds its own database and its own semantics, its much harder to have programs share data.

Yes!

C. A transaction based api would fail anyway occasionally.

D. Note that some OSs (MVS/OS360) that are famous for transactions have the reputation that locking and so-forth are done in the kernel. Which is not actually true.

Abstraction has a cost, and there are no guarantees that if implemented, the resulting performance would be considered acceptable. (It's even possible that many have already tried, and failed.) Of course this doesn't mean that we shouldn't try again and again. But we also have to realize that even if we're successful, this doesn't mean that the solution won't be left completely forgotten, as long as 'good enough' systems exist. (See the other threads for examples.)

Thats fine. Disks are so fast now that I would trade away 50% of the filesystem's performance for correctness in 95% of the software I use. (Eg my code editor, git, compiler, etc). Except I bet in a good implementation the actual performance cost of transactions would be negligible anyway. Especially on top of something like ZFS.

And legacy software can just keep using the current unreliable APIs with all the performance that entails anyway.

>I wince every time I'm playing a video game and it has that "saving - don't turn off your console!" animation.

That seems harsh. Even a transactional FS would have this message. Saving takes time, and the best case scenario for shutting off the machine before it's complete is still data loss. What alternative interface do you propose?

Currently the worst case scenario is that power loss while overwriting the save file leaves the file corrupted and you have to start the game from scratch. How many games do the rename trick correctly when they save? I bet it’s less than half.

The behaviour should be that if power goes out while saving, the game either updated the save file or it didn’t. Saving should be atomic. That’s the best we can do, and what the user should expect if power goes out before it’s finished saving the game.

I'm sad Transactional NTFS is being deprecated. But it has been that way for a few years now and there is no indication it will be actually removed any time soon.

I'm quietly hopeful it will de-depreceted at some point. :)