quickquestion42

Yes, fair enough. Note that I never accused anybody of not telling the truth. My point was about how useful sqlite can be as the posterchild example of a good test suite when we can't see the secret sauce. Not that it doesn't exist. I deleted the other comment because after reading yours I realised it was needlessly controversial. I think you're right in that SQLite can be considered an example of having a good test suite when looked at from a different perspective, just not from the point of it (the test suite) being open source.

EDIT: Deleted a controversial comment about the often praised SQLite test suite not actually being public and about locking behaviour in SQLite. References are [1] and [2].

[1] https://www.sqlite.org/th3.html

[2] https://beets.io/blog/sqlite-nightmare.html

The linked project seems to be a wrapper around

https://github.com/pkg/sftp

Where can I learn more about the rename() trick not being safe?

I was also (incorrectly) assuming that renaming a file after having fsynced the contents is a valid mechanism to perform "atomic" file writes where the write either appears completely at the destination path, creates a zero byte file or does not happen at all. But it should never produce an "incorrect" non-zero byte file - even in the face of crashes.

It is definitely something that is used in the wild so it was surprising to me that it is not correct. I even found a recent LWN article suggesting otherwise, so there seems to be a lot of confusion on this topic.

Would love to learn more if anybody has detailed infos or a link to the relevant mailing list thread or similar.

EDIT: Glibc documentation also implies that the "rename trick" is indeed safe:

From https://www.gnu.org/software/libc/manual/html_node/Renaming-...

> One useful feature of rename is that the meaning of newname changes “atomically” from any previously existing file by that name to its new meaning (i.e., the file that was called oldname). There is no instant at which newname is non-existent “in between” the old meaning and the new meaning. If there is a system crash during the operation, it is possible for both names to still exist; but newname will always be intact if it exists at all.

Another relevant source that suggests the trick is safe is https://lwn.net/Articles/327601/. Keep in mind that Ted Tso is the ext4 maintainer.

> For the longer term, Ted asked: should the above-described fixes become a part of the filesystem policy for Linux? In other words, should application developers be assured that they'll be able to write a file, rename it on top of another file, omit fsync(), and not encounter zero-length files after a crash? The answer turns out to be "yes," but first Ted presented his other long-term ideas.