PostgreSQL's fsync surprise
lwn.net
lwn.net
The fundamental API primitives should be atomic changes. Atomic write (bytes), and atomic append. The funny thing about it is that POSIX already supports basically this API (datagrams) for both IPC and networking. It just doesn't support this API in the one place it would be most useful - the filesystem.
Ideally I want:
- Write() to be blocking / atomic by default. Don't return until data is safely committed.
- A transactional API: begin(fd); write(); write(); err = commit(fd). If any error happens, commit returns the error and none of the data is stored.
- An IOCP-style API for non-blocking applications. This is the API databases want to use, with the loop being <get network request>, <write data to filesystem>, <yield>, <get write completion event>, <send confirmation to client>.
- Deprecate fsync & friends. If you don't want to wait for the data to get committed, write in non-blocking mode and ignore the completion event.
Solving this problem in end-user applications is really hard - almost no applications implement atomic write on top of filesystems correctly. And they shouldn't have to - this should be the job of the filesystem. The filesystem can do this much more safely, with better guarantees, better performance and better error handling. Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. "Do not turn off console while game is saving" is an embarrassment to everyone.
This exists, it's called Linux AIO (distinct from Posix AIO). The problem is, when you use it, you have to reimplement caching (and buffering) in userspace instead of journaling – which is just as hard to get right and can just as easily – maybe even more easily – lead to corruption. (Postgres, as an example, relies on the OS to buffer writes.)
> - Write() to be blocking / atomic by default. Don't return until data is safely committed.
This is the wrong thing for 99% of use cases. You get terrible performance unless you batch things, which means that casual use ends up with severe performance issues. (IIRC this was actually a problem on Android devices – many apps were misusing SQLite by not using transactions, resulting in every single database update being a separate atomic write to disk. Not only did this kill performance, but it caused excessive flash wear.)
> Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone.
Databases (among other systems) need features and control over the journal that a filesystem cannot provide. (Think MVCC, replication, etc.)
Beside – someone has to write the filesystem, and that filesystem uses largely the same mechanisms in the kernel that are exposed to userspace.
I agree fully that fsync ought to be deprecated for something with much more clearly-defined semantics. Both OS X and Linux have made attempts at this (F_FULLSYNC and sync_file_range, respectively), though clearly at least Linux still has some work to do.
But – barring such unclear semantics – the general model of using fsync to guarantee ordering is not a complex one to understand, and matches most use cases well.
We don't really buffer writes via the OS that much, most of them are buffered in postgres' internal buffer cache. We even force the kernel's hand to write out things earlier (otherwise there often are huge latency spikes). It's more reads where it's useful because it allows the OS to cache as much IO data as the current memory situation on the system makes sensible.
> > Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone.
You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks.
We can make that fast by letting the OS / libc batch write transactions. Although if `write()` blocks until the batch commits & gets called in a loop we'd have a performance problem. Non-blocking APIs don't have that problem though.
The guarantee I want is that every write moves the system between two states: (not written)->(written). It should be impossible to reach some third (partially written) state through disk failure or an inappropriate power loss.
Those are two very different things, potentially. If the write traverse a userspace-kernel boundary, there is no way to make that fast for small writes. If the batching happens entirely userspace, but just in a standard implementation, then sure.
But people tend to severely underestimate how expensive syscalls are.
That's not the problem. (OSes already do this.) The problem is one of locality. (Data writes are spread out among many parts of the disk, causing many writes to be issued for any given transaction.) You get around this by (1) logging to a journal, then (2) batching many seconds worth of writes in the hopes that they touch fewer blocks than the sum of their parts (or, in the case of rotary media, that they can be ordered efficiently). You can't just "wait" for this to happen before responding to the client unless you like 5+ second service times.
Which brings us back to the necessity of a journal, which, as I already stated, must be managed by the application because different applications require different features and control which the filesystem doesn't (and likely can't) expose.
> This is the wrong thing for 99% of use cases. You get terrible performance unless you batch things, which means that casual use ends up with severe performance issues.
The right answer would be to batch writes in the OS, but of course this would deadlock without non-blocking APIs. Maybe you're right - maybe write() should have the effectively non-blocking API it has now with fsync as a commit. But I still want my guarantee that if the system / hardware dies halfway through a write (or batch of writes) then data won't be left in a corrupt half-written state. Package installations and OS upgrades should be able to do the entirety of their work in a filesystem-level transaction. I shouldn't need to worry about my OS needing a reinstall because my laptop ran out of power halfway through an update. I just can't think of any sensible way to do this using the APIs we have today.
> Beside – someone has to write the filesystem, and that filesystem uses largely the same mechanisms in the kernel that are exposed to userspace.
But importantly not the same mechanisms. The OS has control over write ordering, and most block devices support read & write barriers. Filesystems lean heavily on these primitives for correctness - and for good reason; they're really useful primitives for building correct, performant systems. But for some reason they aren't exposed to userspace. Userspace applications are instead left scratching in the dirt with fsync. It would be much easier to build correct, performant databases in userspace if we had access to this stuff.
> most block devices support read & write barriers
No, they really don't. Some hardware devices might. I'm not aware of any SAN that does. (I've specifically asked this question of the developers of three.)
> Filesystems lean heavily on these primitives for correctness
In Linux, not since 2010 they don't [1], for exactly the reason that they're poorly implemented by devices.
https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
...but sadly, they seem to be on the path of being deprecated by Microsoft...
https://msdn.microsoft.com/en-us/library/windows/desktop/hh8...
https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob...
everything you describe will murder IO performance because you're essentially implementing a database with MVCC. it makes much more sense to implement what you described as a VFS library.
(In fact, ZFS with `sync=always` set on the dataset would do exactly what the parent commenter is asking for without requiring a whole new API. If someone really cared about performance they could even make this perform well with a battery-backed DRAM device for their intent log... but by that point we're way outside of the discussion of a general-purpose filesystem. If you really need guaranteed consistency for every write _without_ crippling performance, ZFS can do it for you at a hardware cost. Writes get cached in RAM, batched, flushed in transaction groups, and the fast/durable SLOG protects against corruption or loss of power or device failure without harming performance much. $$$ though. You'd better have quite a bit of RAM, and need a proper enterprise-grade DRAM storage device - or probably two in a mirror if your data is this critical).
And for every case of something that would have lower IO performance with atomic appends, I'm sure we can think of ways the kernel could make those atomic writes fast. Lets say you're copying a big file by wrapping a transaction around a read();write() loop. The OS could make the transaction extremely cheap by simply not changing the file's INODE list (or whatever) until the write has completed. In a filesystem like ZFS, using a transaction for this would let ZFS not update its filesystem checksums until the copy has finished, which would probably in aggregate result in improved performance.
Obviously this would be horrible for PG to use as a replacement for fopen() and other systems, but for many, many use-cases SQLite is a good replacement for fopen().
As in, "Obviously [it] would be horrible for [postgres to be used] as a replacement for fopen() and other systems, but for many, many use-cases SQLite is a good replacement for fopen()"
"Obviously [SQLite] would be horrible for postgres to use as a replacement for fopen() and other systems, but for many, many use-cases SQLite is a good replacement for fopen()"
Gradi / Ghredh is also the same root used in words like transgress, degrade, gradual, centigrade, congress, etc.
- Write() to be blocking / atomic by default. Don't return until data is safely committed."
How do you deal with user pulling out usb stick?
Asynchrony is absolutely critical for performance.
Filesystem I/O, and, really, disk/SSD I/O is really very much the same as network I/O nowadays. You might be traversing a SAN, for example, or the seek latency on HDD could be reasonable for an HDD but nowadays that's just way too high anyways.
Transactional APIs are very difficult for filesystems. Barriers are enough for many apps. A barrier is also much easier to start using in existing apps. You'd call a system call to request a write barrier, and wait, preferably asynchronously, for completion notice (whereupon you'd know all writes done before the barrier must have reached stable storage).
Async QIO style IO is all good clean fun, until someone writes something like:
fml(...) {
char buf[1024];
sys$qio(QIO_READ, buf, sizeof(buf));
return;
}
main() {
fml();
do_stuff(); //surprise package delivered into stack here!
}As I recall, older operating systems like VMS and MULTICS all did this, and it was all tremendously complex. Unix’ simple stream-of-bytes file abstraction was a reaction to this, and it worked so well that it became the prevailing model. Before doing again what didn’t work before, check up on why it failed the previous time.
Others have mentioned SQLite as replacement for fopen()... that’s a fairly RMS-like experience on modern systems
(I wonder if this is being downvoted because people think RMS means Stallman)
Unix made for very simple filesystem internals, with a crappy interface (worse is better). RMS was a very complicated filesystem with a nice interface (the right thing).
Hummmph, it was kind of a pleasure on a VAX, but on a PDP-11 under RSX or RSTS/E the libraries for it ate so much of the (16 bit) address space that it became a real challenge to fit a useful application into memory.
Also, it only sort of supported streams, they were not really byte streams like Unix, but variable length record streams with CRLF marking the record delimiters. So porting Unix code to VMS was never quite as straightforward as one hoped.
We did a nice P/Invoke wrapper around native Win32 functions in C#, exposing FileStream and such, that the rest of the code could then easily consume.
I think the main complexity is that there are no "official" high-level wrappers the way there are for non-transacted APIs. But at the level of Win32, there is no significant difference.
I worry that if the Postgres people do make that change, they'll find themselves hearing from a different set of kernel developers that they should have known direct IO doesn't work properly and they should be using buffered IO instead.
Previously I'd thought the latter was the general view from the kernel side.
For example this message from ten years ago, and other strongly-worded views in that thread: https://lkml.org/lkml/2007/1/10/235
In particular, I'd taken this bit as a suggestion that if people found problems with buffered IO then the right thing to do is to ask the kernel side to improve things, rather than switch:
« As a result, our madvise and/or posix_fadvise interfaces may not be all that strong, because people sadly don't use them that much. It's a sad example of a totally broken interface (O_DIRECT) resulting in better interfaces not getting used, and then not getting as much development effort put into them. »
That definitely will happen. But the fact remains that at the moment you'll get considerably higher performance when expertly using O_DIRECT, and there's nothing on the horizon to change that.
> For example this message from ten years ago, and other strongly-worded views in that thread: https://lkml.org/lkml/2007/1/10/235
> In particular, I'd taken this bit as a suggestion that if people found problems with buffered IO then the right thing to do is to ask the kernel side to improve things, rather than switch:
I think partially that's just been overtaken by reality. A database is guaranteed to need its own buffer pool and you're a) going to have more information about recency in there b) the OS caching adds a good chunk of additional overhead. With buffered IO we (PostgreSQL) already had to add code to manage e.g. the amount of dirty data caching the OS does. The only reason DIO isn't always going to be beneficial after doing the necessary architectural improvements, is that the OS buffer pool is more adaptive in mixed use / not as well tuned databases.
In my opinion, even a careful reading of the fsync man page does not cover what exactly happens if you close an fd, reopen the file in another process, and then call fsync. Am I supposed to read kernel source code? Ideally, after reading a man page, I should have no questions about exactly what guarantees are provided by an API.
I'm certainly not qualified to criticize anyone for the current situation, and, as the article points out, even some of the more egregious sounding behavior (marking as clean pages after writing fails) has a pretty reasonable explanation. But, IIRC, as storage capacities continue to rise, error rates aren't falling nearly as fast. So, I'm left kinda wondering if there is some day in the future where the likely hood of encountering an error finally gets high enough that things don't work pretty well anymore.
Perhaps we need database and backup oriented filesystems and not funny looking files on top of generic filesystems.
On top of that, many issues you may be facing re. files now have already been resolved if you change the stack: you can't do transactions with fs.
There are some additional performance and behavioral advantages to working with the storage devices directly. Anecdotally, if you run databases on virtual machines (never recommended but many people do), using raw block devices instead of a file system often seems to eliminate much of the disk I/O weirdness that occurs under VMs.
What was surprising is that the direct I/O behavior appears to be conditional on whether you are accessing the storage through a file system. My database kernel is block device agnostic, using files and raw devices interchangeably via direct I/O. Against expectations, when we accessed the same virtualized storage as raw block devices, the behavior was like bare metal even though we are running the exact same operations over the same direct I/O interface in a VM. Basically, the only difference was the file descriptor type.
I'm guessing that file systems are virtualization aware to some extent and access through them is actively managed; raw device accesses are VM oblivious and simply passed through by the storage virtualization layer.
e.g. in mysql: https://dev.mysql.com/doc/refman/8.0/en/innodb-raw-devices.h...
Proper handling and reporting of hardware-level errors all the way up through the stack (driver, block layer, filesystem, C library) to the application so it can recover in a reliable way is not a simple operation!
Simple operations are open/close/read/write. Those work. Until they don't, then you need to know how far back the operations you already did and "assumed" had worked didn't. And in this case the promise made to PostgreSQL by fsync() wasn't as firm as the "obvious" interpretation of the documentation would lead one to believe.
That behavior seems problematic.
As always, there's a great Dan Luu blog post on the subject: https://danluu.com/filesystem-errors/
> Filesystem error handling seems to have improved. Reporting an error on a pwrite if the block device reports an error is perhaps the most basic error propagation a robust filesystem should do; few filesystems reported that error correctly in 2005. Today, most filesystems will correctly report an error when the simplest possible error condition that doesn’t involve the entire drive being dead occurs if there are no complicating factors.
Emphasis added.
If the storage device has disappeared for good, you can just return EIO for all further I/O operations, and mark all open file descriptions for which there were dirty pages such that any further fsync() calls on the corresponding fds return an error?
I mean, either you think you can still retry, then you should keep the dirty page around, or you think retrying is futile, then feel free to drop the dirty pages, but make sure anyone who tries something that would make this loss visible gets an error, which should only require keeping flags on open file descriptions, and possibly pages/inodes/block devices that (semi-)persist the error at the desired resolution, which you can broaden if the bookkeeping uses too much memory.
The memory can be reclaimed when 'umount --force', or something like that, discards filesystem dirty state.
fcntl(fd, F_CLR_FKNG_ERR);That's not exactly true. In the thin-provisioned block device case, administrator action can make it resume accepting writes.
Comfort is of little consolation or use if the operating system is this unreliable, especially since making sure that data is safely and correctly stored is core, basic functionality.
wondering what happens to other critical libraries such as RocksDB/LevelDB, what actually happens when there is hardware error not limited to unplugged usb cable?
[1] My last experience was due to bug where certain pattern of data made MySQL/MariaDB think the data page was encrypted, after which it proceeded to discard that page and crash complaining that data is corrupter and from that point on refused to start until data got restored.
Also, perhaps the error flag should keep the inode in core. Note that the pages would still get thrown out, so as far as memory pressure this is not the end of the world.
And there in lies the rub. Sybase ASE calls fsync() upon every commit, which is the reason that database devices are still mostly implemented with raw devices. Before version 11.9.2 (as far as I recall) you ran the exact same risk if you used the file system as devices. Now it's safe, but performance can get pretty heinous on write intensive systems.
Those are journal commits, not the commits that the piece you quote is talking about (actual data files).
No, that is a "recipe for disaster", as they say. Not doing something that everyone acknowledging as important, because that's a "lot of work" is what makes projects a mess. I've seen that many times on various projects.
And it'd not be the default anyway, as it requires more tuning.
We are working on getting there.
You sound like a Dilbert manager's "I don't care, just have it on my desk at all costs by Friday."
That is not sound engineering.