Comment by loeg | Hacker News Reader

Comment by loeg | Hacker News Reader

Parent Full thread

Try "ls -f" (don't sort)?

Which filesystem you use will also make a big difference here. You could imagine some filesystem that uses the getdirentries(2) binary format for dirents, and that could literally memcpy cached directory pages for a syscall. In FreeBSD, UFS gets somewhat close, but 'struct direct' differs from the ABI 'struct dirent'. And the FS attempts to validate the disk format, too.

FWIW, FreeBSD uses 4kB (x86 system page size) where glibc uses 32kB in this article[1]. To the extent libc is actually the problem (I'm not confident of that yet based on the article), this will be worse than glibc's larger buffer.

[1]: https://github.com/freebsd/freebsd-src/blob/main/lib/libc/ge...

with "ls -f" on 1.9M files its 45 seconds, much better than regular ls (and system load of 94)

2.25M and its 60 seconds

I'm also spamming about 16-18 thousand new files per second to disk using a very inefficient set of csh scripts...

A more efficient one-liner:

    seq 1 8000000 | xargs touch

Interestingly, that didn't work.

Strangely "jot -w file 8000000 1 | xargs touch" worked.

I'm going to try to replicate this because I sense some kind of tomfoolery.

The command I quoted works verbatim on one of my Ubuntu systems. It's ~60X faster than eg "for i in $(seq 1 8000000); do touch $i; done" because it creates many files per fork+exec, and fork+exec is a much heavier operation than creating an empty file.

Seq isn’t installed by default in FreeBSD, while jot is. That’s all. I think seq may be part of coreutils and installed as ‘gseq’? Not entirely sure.

seq is in /usr/bin, its default in FreeBSD as on FreeBSD 9.

I'm actually not sure why it failed after generating 1.3M files, no error messages or anything, it was weird. Initially I thought maybe it was like an inode/fd issue or something but no.

Reply on news.ycombinator.com