Hacker News Reader

Hacker News

Comment by exdsq | Hacker News Reader

Parent Full thread

exdsq

Can you mv niceDir/* newNiceDir/ && rm -fr niceDir to reset the directory size?

View on HN

I'd probably

1) hardlink all the files from the old directory into a temporary new directory, and

2) rename the new directory to the old directory's name.

I'm no Unix guru but I believe that this should be completely unintrusive to even running processes, or at least less intrusive than your approach. Also failing in the middle seems more graceful.

> completely unintrusive to even running processes

Unless between the time that you create the hardlinks and the time that you delete the old directory, a process creates new files in the old directory. Unlikely of course, but not impossible.

Yep, I've already thought of it in the several minutes after posting it. If it were possible to make a hardlink to a directory, my solution for a directory with a varying set of files would be:

1) hardlink all the files from the old directory ("foobar") into a temporary new directory ("foobar_tmp1"),

2) create a hardlink ("foobar_tmp2") to the old directory ("foobar"),

3) rename the new directory ("foobar_tmp1") to the old directory's name ("foobar"),

4) patch the "old" directory ("foobar" now linking to the shrunk directory) with the new files in "foobar_tmp2",

5) get rid of "foobar_tmp2".

Still doesn't take care of all possible issues such as a program creating a file in "foobar" between steps 2) and 3) and reopening it very quickly after step 3), but at least it seems to avoid the issue of lost files.

Sadly step 2) seems impossible. I still believe the original version should be transparent for a directory with a fixed set of files, but of course very large directories probably have a higher chance on average of their set of files changing in a short period of time. That may or may not be a problem.

EDIT: Maybe a mandatory lock on the directory around the original step 2) with an additional check whether the directory's file set didn't change would work on Linux, if such a thing works the way I think it does (quite possibly it doesn't).

You can't hardlink directories on Linux.

They seem well aware of that :p

I assume that if you could create hardlinks to directories, it would be to the same directory file and thereby point to the same garbage you want to get rid of. Don’t do that and you have a symlink.

No, the point is to preserve the original set of files to account for lost file creations after the directory rename. What point would a symlink achieve? The original directory (the i-node, NOT the entry in the parent directory) with the new, unaccounted-for files would be gone and the symlink would point to a directory with only those files that would have been already preserved by the rename. So pointing "to the same old garbage" is exactly the point.

You can't rename over a non-empty directory, so you can't do this atomically without using a symlink.

Oh crap, I've never noticed that (or maybe I did many years ago and promptly ignored that because it wasn't what I was looking for at that time). Well, there goes the whole idea. Symlinks are of no use here, or at least I don't see how they help.

If you know in advance that you'll need to switch a directory atomically, you can make `dir` a symlink to `dir_v1`. Then you can atomically replace it with a symlink to `dir_v2`.

This doesn't really solve the situation we're discussing though.

Wouldn't you need a step 1.5: rm -rf <old directory>

Well, apparently I would, which mildly sucks because it takes too much time that I wanted to avoid completely by the ole' atomic rename technique. Worst case, at least one could possibly rename the old directory to a temporary name and then immediately rename the new directory with the new file links to the old name and hope for the best. Sadly the atomic technique doesn't seem to work here. I never noticed that I can't do this with a directory. Well, shit.

On top of what others said, creating a copy of an existing directory isn’t trivial. You have to copy

  - owner and group,
  - rwx flags,
  - OS-specific flags such as the FreeBSD immutable ones,
  - ACLs,
  - extended file attributes
  - possibly stuff I forgot

Some of these you should set before copying in the files for security reasons, some of these you can only set after doing that.

If the directory isn’t writable, for example, you can’t simply create it writable, copy in the files, and make the directory read only, as that opens a hole where others can write files to the directory.

I think the only safe way is to create a directory that’s only readable and writable by you, copy in the files, then set the attributes right.

Reply on news.ycombinator.com