How to change symlinks atomically (2005)
blog.moertel.com
blog.moertel.com
May not be available for your operating system variant. May not be available for your file system. Not applicable to some remote file systems. May not function properly in some virtual machines. Consult your storage area network vendor for additional information.
The only reason I suspect things do unlink-then-rename is that (evidently) rename() doesn't promise anything about what happens to the "src" item if rename() fails, just that something will still be in "dest".[1]
But I don't know why the commands do something different, and don't want to try reading the source of either GNU or BSD mv to find out why.
[1] - http://pubs.opengroup.org/onlinepubs/009695399/functions/ren...
The new WSL does have symbolic links, but these are only available to Linux programs.
An application should not change the default security policy.
When someone uses the term "symbolic link" to mean something different than someone else using the term "symbolic link", we can assume they:
(a) do not understand what UNIX is calling a symbolic link, either because they have never heard the term, did not research it properly, or have some kind of learning disability
(b) are attempting to intentionally confuse users
We do not assume that "symbolic link ``features''" now means something else: We need justification to do that, and we don't have it:
A "link" is what POSIX calls a directory entry (§3.130), and a "symbolic link" is simply a directory entry (link) without a file associated with it.
When we are thinking about this clearly we can see what Microsoft did wrong, but we're under no obligation to support them in their definition because clearly their definition is less useful than the POSIX one.
> Some Linux distro could implement the same policy,
Users would revolt; they did revolt.
SELinux was popularized by a number of Linux distributions, and included a configuration that did exactly this, however people just turned it off because breaking applications pisses people off.
> but symlinks would still be symlinks.
No.
Just because they use the same words does not make them the same thing. You can use α-conversion to fix this without anyone else's help, and call Microsoft's "symbolic link" foo and UNIX's symbolic link bar, then you can find the sentence "foo would still be bar" the nonsense that it is.
All the other still apply, and the GP forgot to tell you to check your HD model's documentation.
If it doesn't exist in practice or does something different in a thousand edge cases, it's rightfully dismissed as nonexistant.
I thought that requirement was from POSIX.
Does xfs not conform to POSIX?
The way I understand the article's proposal is this:
1. Create new symlink pointing to desired file (assumed to already exist in a stable state).
2. Move new symlink over old symlink.
They exist as directory entries with a small in-directory content only, thus syncing the directory they exist in is sufficient to persist them.
See e.g. https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Symb...
I think it is more useful to think of the symbolic link as "inside" the directory it is in, because this tells the programmer what to do (and what to sync: the directory).
[1]: http://pubs.opengroup.org/onlinepubs/009695399/functions/rea...
A directory, historically (on systems with 14- or 30-character file name limits like we had in the 1990s, at least 4.2 or 4.3 BSD-derived), consisted of a file containing a list of (16-bit inode number, file name) structs, thus making a nice round 16 or 32 bytes per directory entry, evenly dividing a 512-byte block. On some older systems you could see this structure by opening and reading a directory with open and read (I think every system everywhere prohibited writes to a directory because modifying a directory would allow you to access files in unreadable directories, uncleanly delete any file given its inode number, etc). The only special thing about a directory on disk was that its inode said it was a dir, so the filesystem code would trust it as a source of inode numbers and otherwise use it as a dir.
Some of the 14-byte-name systems supported symlinks. As there was no space in the directory entry for anything other than an inode number, the link contents would have to be accessed through an inode. (I think I've heard of systems that could store very small file contents like typical symlink target names in the inode instead of allocating a data block, but can't name one).
The number you see in ls -i, which is sometimes called just the "inode" (and apparently also called the "file serial number"), is the number in the directory entry that's used to find the inode. I guess someone somewhere had or intended to have a system that could store symlinks in the directory instead of consuming an inode.
The fact that it does or does not should actually be in the documentation IMHO; probably a good enhancement request.
POSIX does (for some strange reason) permit a symbolic link to have an "inode" (well, d_ino), but there is no way to open this "inode", and no UNIX implementations to my knowledge do this.
HOWEVER, looking into it it seems that the target may actually be stored in the inode. This mean the symlink, though a file, has no contents and thus requires no fsync. Does this sound about right?
No they don't.
First of all, POSIX doesn't use the term "inode number" but "file serial number" which is silly wankery since there's no way to access a file using its "file serial number". They might as well say it was "implementation defined".
Secondly, POSIX used to specify[1] that their inode was "unspecified" because UNIX systems don't store symbolic links in separate inodes, but in the directory that contains them. Now, POSIX specifies symbolic links do have a "file serial number"[2], which could be implemented as a separate file block (confusingly Linux calls this an "inode" which has nothing to do with what UNIX called an inode -- a better term might be "virtual inode" but Linux uses that for something else entirely...)
To this end, I think the only sensible interpretation is the original one: UNIX symbolic links don't have inodes, and POSIX symbolic links might as well have a "hash" of the file contents in the "file serial number" since you can't do anything with it anyway.
[1]: http://pubs.opengroup.org/onlinepubs/009695399/functions/rea...
[2]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/re...
> HOWEVER, looking into it it seems that the target may actually be stored in the inode. This mean the symlink, though a file, has no contents and thus requires no fsync. Does this sound about right?
You have to fsync() the directory that contains the symlink.
What UNIX called an inode is unfortunately not what Linux now calls an inode. That thing might have better names, but we are stuck with it.
I don't know if it is useful to say "Linux inode" because that can refer to both what "ext4" calls an inode and what the Linux kernel refers to as a "virtual inode".
What POSIX called a "file serial number" (and goes in the `d_ino` field) is also not what Linux calls an inode. I sometimes think this is okay to refer to as a "POSIX inode" because POSIX doesn't otherwise use the term, and up until SUS6 the POSIX file serial number was compatible with a UNIX inode.
However I'm not trying to nit: Trying to develop a simplified mental model is essential to determining what to do (and what to sync!). From an application point of view, knowing that a symbolic link has a name (and contents) and lives in a directory, can help guide us.
How so? Here's a v6 on-disk inode: https://github.com/hephaex/unix-v6/blob/master/ino.h
and here's an ext4 on-disk inode: http://lxr.free-electrons.com/source/fs/ext4/ext4.h#L704
The latter is considerably larger, to be sure, but still contains all of the fields from the former (many of them with the same name!).
UNIX called them "inode" and "permanent inode"; Linux calls them "in-memory inode" and "on-disk inode". The BSDs use "vnode" and "inode".
Edit: If you want to fsync the symlink you can do an open with nofollow iirc (on my phone so I can't check) so you get the actual symlink and not its target.
I think this is true in many filesytems, not just `xfs`.
If your atomicity requirement you never want the file to disappear from the POV of an external process, then the OP's method is sufficient. If you want crash-proofing as well, then you will need an fsync() -- preferrably on the tempfile BEFORE the rename().
d=open("."), unlink("t");
/* 1 */ symlink("new","t");
/* 2 */ rename("t", "link");
/* 3 */ fsync(d);
/* 4 */ close(d);
You can insert a second fsync() where you suggest at point (2), but all this will guarantee is that we will have "t" in the directory because the symlink contents are part of the directory they live in. This might be useful for some applications, but the cost of two disk writes is high enough it may be worth redesigning your application.
Now on modern filesystems, a non-huge symlink will be stored in the inode itself and presumably enjoys some sort of atomicity. But there is nothing in the standard about that.
No, I don't think you can, bugs notwithstanding. A "link" (§3.130) is what POSIX calls a directory entry.
> But there is nothing in the standard about that.
The "standard" (POSIX) doesn't talk much about crashing, however if mkdir("a") could destroy "b" – even during a system crash (§3.387), then users would complain.
In Unix we have this convention that `rename()` is our atomicity swiss-army knife. I guess it makes it easier for implementers to only have to make their strong guarantees for one system call (+ a few related friends like `renameat`).
Now you could argue that `rename()` is a bit too under-powered for this job, and maybe we want transactions or something. But NTFS tried that too and deprecated them.
What I would like to see in Unix is better support for anonymous files and directories. You could use this with things like `renameat()` so that the temporary never touches the filesystem and is automatically cleaned up if the rename fails.
Exactly. This approach makes the various calls orthogonal. Various syscalls create anonymous reference-counted inodes on a filesystem (which disappear when all references to them do), and syscalls like linkat or renameat or installs such an inode (referenced by file descriptor) into the filesystem with a name.
O_TMPFILE allows creating an anonymous file inode. Ideally, a syscall would exist to do the same for an anonymous symlink inode.
With a file descriptor you can create multiple mappings to the same physical address -- something otherwise impossible on UNIX or Linux (although mach allows you to vm_remap[1] which is often sufficient).
[1]: http://web.mit.edu/darwin/src/modules/xnu/osfmk/man/vm_remap...
It creates both; memfd_create (and timerfd_create, signalfd, eventfd, userfaultfd, epoll, and others) use a kernel subsystem called "anon_inode" to create an in-memory inode to back the file descriptor.
[1]: http://lxr.free-electrons.com/source/mm/shmem.c?v=4.2#L2949 [2]: http://lxr.free-electrons.com/source/fs/ramfs/inode.c?v=4.2#...
The hard bit is that for it to be useful, you would need atomic renames from the TMPFS to some other arbitrary filesystem.
I guess abolishing the EXDEV error would be possible where the target filesystem has journaling, but it sounds like a big job to me.