Hacker News Reader

Your response pretty aptly demonstrates the problem in the original article.

If you write to a temp file and then use rename, you need to pick a suitable temporary name (without a race condition) that other programs will ignore, and now you've probably created a garbage file if the program crashes before you get to the rename. You also need to make sure you've preserved the permissions in the original file, and hope that nobody else is using the same trick to update the same file.

There are API's and standard tricks for these things too. But it's difficult to be sure that you've covered all the corner cases and impossible to be sure that everyone else has covered all the corner places.

This all adds complexity compared to having a simple, standard, guaranteed way to atomically update a file and makes it less likely that most programs will do it right.

As I said, if it's too unbearable a hack for you to unlink() the file directly after creat()ing it (to get an anonymous file handle that nobody else can see), for the unlikely case that the system crashes just in between, then just go ahead and use O_TMPFILE. It was made for you!

If you don't want to use O_TMPFILE (not super-portable) then just open(path, O_CREAT|O_EXCL). That fails should there be another file lying around, and you can try again with another random name.

Btw. you can also just mmap(2) some anonymous non-disk-backed memory, and possibly open_memstream(3) on it to get a FILE * if that suits you better.

Even easier than open(2) with a random file name is mkstemp(3).

Exactly. It takes a whole collection of workarounds to get well behaved file semantics across crashes, and the workarounds are different for Linux, BSD, and various other UNIX variants, and may differ depending on the file system type and system version.

Don't know why you're downvoted. Everything you said is correct.

Additionally, in Linux, you can get asynchronous fdatasync(2) with sync_file_range(2).

aio(7) is weird. It really only works with direct I/O (which is necessary much less often than people think it is), and IIRC aio_fsync(2) isn't implemented on Linux, but that doesn't matter so much because generally direct I/O implies synchronous I/O (but not always). See http://lse.sourceforge.net/io/aio.html

AIO does not require O_DIRECT. Most Linux file systems do not implement aio_fsync because they put the writes into the page cache and block until it is flushed to disk before signaling completion. That means aio_fsync can be implemented by no-op.

Unfortunately that is not documented in any man page I've read, because that is very useful (crucial) to know and not at all obvious. I reiterate that aio is weird :)

Also, I'm pretty sure that issuing IOCB_CMD_FSYNC from io_submit(2) has given me an error in the past. Not sure about aio_fsync(2), because last time I checked, POSIX aio only had a pthread-based backend, rather than the native support that libaio (i.e. io_submit(2)) provides.

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.)

O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly.

How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.

"fsync" synchs the whole file, not just one I/O, which can take a while. Databases such as MySQL's InnoDB, which puts multiple tables in one file, can have independent I/O going on in different parts of a file.

aio(7) has the right mechanism, a callback/signal on completion. But it's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.

The original article complains that UNIX/Linux file system semantics aren't well enough defined for database safety. He's right. They're close, but not quite there, because behavior after a crash is unspecified.

sync_file_range(2) allows you to specify ranges to sync, instead of the whole file, but it's not that useful anyway.

In most DBs, you have separate data and log. The log is always only appended to, and the nature of disks is such that grouping concurrent transactions together into a single commit is a big win regardless of how syncing works. So syncing all outstanding I/O to the log file is generally what you want to do anyway.

While the data store is always being written to, its writes do not need to be synced to disk until a log checkpoint is made (and the old log subsequently discarded). And since that is an intermittent asynchronous process, there is not much to gain from finer-grained syncing here either.

See e.g. http://www.postgresql.org/message-id/4A51CB76.5020407@enterp...

Actually, we made postgres use sync_file_range() for checkpointing if available in 9.6. Not for durability - there's a few to many caveats in the manpage - but to control how much work a later fsync() has to do. In many workloads checkpointing can generate a lot of writes, and the OS's writeback caching of those can generate a lot of dirty buffers in the kernel's page cache. If the kernel decides to flush those (on it's own or due to an fsync), latency for every other FS operation can skyrocket. We've seen stalls in the 10s of minutes. So we now regularly sync_file_range(SYNC_FILE_RANGE_WRITE), unless the feature is disabled of course, to control how much dirty data the kernel has.

See http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;... and http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;...

Thanks for the info, as a heavy user of Postgres it's great to know that option will be available!

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write.

Note that Linux ext4 does not do this. On a power outage, you can get bogus trailing zeros on a file which you were appending, because the file size was updated before the data was written. I asked Ted T'so about this and he said it was working as intended.

https://plus.google.com/+KentonVarda/posts/JDwHfAiLGNQ

ZFS does do this. Changes are atomic, so they either happen or do not. There is no in-between.

I don't know about ZFS, but the usual write(2) API does not support this: It might for example return early with a short write because some interrupt occurred. Can happen for all "slow devices" (see signal(7)). And I think that's a good thing and am sure many programs expect this.

The signal(7) man page also states clearly that a local disk is not a "slow" device, so this seems moot.

    read(2), readv(2), write(2), writev(2), and ioctl(2) calls on "slow" devices.  A "slow" device is one where the I/O call  may  block  for  an
    indefinite  time, for example, a terminal, pipe, or socket.  If an I/O call on a slow device has already transferred some data by the time it
    is interrupted by a signal handler, then the call will return a success status (normally, the number of  bytes  transferred).   Note  that  a
    (local) disk is not a slow device according to this definition; I/O operations on disk devices are not interrupted by signals.

oops, had that wrong. Thanks for noticing!

It does happen with the default setting, but if you journal everything, then it doesn't happen. Performance on spinning rust, however, drops 50%.

Oh! So that's where all the random nulls in my shell history came from. Zsh would sometimes complain about a corrupted history after a bad crash, and I always wondered what caused maybe 20 nulls to be added to the end of the file. Thanks!

> NTFS doesn't do append correctly.

Care to elaborate on that? I haven't seen issues with the two mechanisms I know of in NT:

1. CreateFile with FILE_APPEND_DATA

2. WriteFile with the OVERLAPPED offset set to ~0

Both of these get you writes that are as far as I know safe to have multiple processes writing to a common log file. Not sure about flushes to disk but you could probably get away with FlushFileBuffers for this.

NTFS handles append just fine. As another comment speculated, the parent may have meant NFS, which does notoriously have lots of problems with append, and many other POSIX semantics.

Sorry, meant NFS.

> On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position.

I would never expect that. It's not how it works. But what has to work is the dup() case, or the inherited descriptors case like

$ ( do_this & do_that & } > /some/file

because here they share the same file descriptor (as opposed to having only the same file opened on stdout). Note this still doesn't guarantee that writes won't be interleaved. I think there is only a guarantee that writes up to 512 bytes won't be interleaved. The answer here is: Don't do simultaneous writes. Or implement your own synchronization mechanism. It's just not a requirement with a universally satisfying solution that should be implemented in the kernel.

> NTFS doesn't do append correctly.

If you meant NFS: It deliberately violates POSIX file system semantics. I guess it's just too hard to implement. For example, an NFS server can't know whether the machine which owned a lock crashed or is just temporarily disconnected from the network (and has still a process running thinking it owns a lock and not knowing the server is wondering).

In any case, not the API to blame.

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.

How could we possibly implement these "transactions" if the write is _not_ at the end of some file?

> It's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.

What the disk does is in the end controlled only by the disk itself (and of course environmental forces). So it's always best effort.

> Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.)

It seems O_TMPFILE cannot be used to atomically replace existing files either. If only one could use renameat2() on a raw fd or force linkat() to an atomic overwrite.

Rename doesn't always do what we want. The one serious investigation I've seen [1] unhelpfully states that "[W]hen a file is appended, and then renamed ... many file systems recognize it and allocate blocks immediately." Suggesting that some file systems don't, but failing to name them. In general, appends followed by directory operations are not order-preserved.

[1] https://www.usenix.org/system/files/conference/osdi14/osdi14...

There's no order preserving guarantee. Code should do

    fd <- open(tmpfile)
    write(fd, data)
    fsync(fd)
    rename(tmpfile, finalpath)
    fsync(parent(finalpath))

(Strictly speaking there is a problem in the rename because we can't be sure "tmpfile" still refers to the same inode. It would be nice to have a way to link fds, but I'm not aware of any).

According to the paper, existing file systems recognize the (open,) write, rename pattern and don't reorder write and rename. That would break too much existing code. But these are filesystem-specific design decisions.

Btw. it's just the same with CPUs writing to memory. Generally writes can be reordered if the behaviour is not visible to the executing thread. However if there are multiple threads you might need to insert (CPU-specific) memory barriers to prevent reorderings that are visible to other threads. x86 is like those file systems here in that most incorrect code still works, because x86 makes some ordering guarantees.

rename(2) atomic replacement only means atomic with respect to processes running on the system at the same time: anyone looking up the newpath will get either the old file or the new file (and won't, for example, see a spurious ENOENT).

It doesn't mean atomic with respect to a system crash.

The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.

> The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.

However you would need to create a new thread for each simultaneous asynchronous fsync to get the benefits of asynchronicity.