Hacker News Reader

Comment by jeffbe | Hacker News Reader

ext4 seems always to be the best.. no?

Depends on the use case, but ext4 is a pretty good default. I like using IBM's JFS on my laptops to prolong battery life since it minimizes CPU usage [0][1][2].

[0] http://fsbench.netnation.com/

[1] https://www.debian-administration.org/article/388/Filesystem...

[2] http://linuxgazette.net/122/TWDT.html#piszcz

those benchmarks are 10+ years old - are you sure they are still valid?

ext4 is really good at the single user (single process / single thread) use case.

Other file systems shine in different categories like number of features (btrfs) snapshots (btrfs) very large devices (xfs) really high concurrency (xfs).

ext4 has it's place - and deservedly so. But picking the right file system really depends on your requirements. For example if you're building a storage server then ZFS would be a better fit.

XFS is also worth taking notice of since the benchmarks I've read rate it for having faster read and write speeds than ext4. However I don't have extensive first hand experience running XFS (something which I'm currently addressing).

I've set up several hundred storage servers with XFS (up to 250 TB per FS) in the past 15 years. There were sour times, 10 years ago or more, but in recent times it's been rock solid and beat the crap out of other FS. Notice that XFS always hated crappy hardware, because it pushes it to the limits.

Unsurprising given it was built by the same people who built these:

https://www.sgi.co.jp/features/2001/dec/fleet_numerical/imag...

And more recently this:

https://www.sgi.com/pdfs/4555.pdf

Because pushing the envelope of HPC awesomeness was just another day in the office for folks at SGI. :)

XFS is hands down the best at sustained write speeds. Each time I've stacked it up against the competition it wins.

I'm running XFS on a couple of my disks. It seems quick and reliable so far. My only complaint is that you can expand a volume (online, too), but you can't shrink it. Shrinking volumes comes in handy occasionally, and when you need it, you need it. But so far it hasn't bitten me yet.

until you get to many small files (multiple terabytes) or very large volumes.

then it's metadata overheads consume more I/O than the data itself.

Can you elaborate on this or provide a pointer?

I'm storing a lot of small files on an ext4 drive (on a 48 GB Linode). Right now it's 5M files, and they take up 40G or so. I formatted it to use at least twice as many inodes as the default (otherwise I would have run out), and then lowered the block size to 2K rather than 4K, which reduced a ton of wasted space because the files are so small.

That's not terabytes, but it's definitely a lot of small files for the size. I probably will end up using 50M or so files up from 5M (on a bigger drive).

So is ext4 less inefficient for this use case than other file systems? Why? I hear you can turn on options like dir_index but I haven't tried it yet.

Note that since linux 3.8 [1] ext4 can embed small files into inodes thus using less space and allow faster reads. Using that feature and adjusting inode size you can manipulate which file sizes are inlined.

[1] http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd...

OK interesting, so this capability is on by default then? I didn't change the default inode size, but it looks like I could SAVE space by increasing it (from 256 or something). Since I have 1.91M files less than 2048 bytes, I guess I could increase it enough to store most/many of those.

And I guess it halves the IOPS for serving those files, since with a cold cache you need to read block holding the inode and block holding the data.

Happy to. :)

Talk by dchinner @ Redhat https://www.youtube.com/watch?v=FegjLbCnoBw

> Can you elaborate on this or provide a pointer?

In my experience ext4 becomes very unhappy/slow when you have many files/folders in the same folder (I noticed this with backuppc). dir_index may have helped me with this.

Do you have a reason for keeping them stored as lots of very small files in a filesystem?

If not, you might take a look at this page on the Sqlite home site: https://www.sqlite.org/intern-v-extern-blob.html and consider storing the small items within a sqlite database file instead.

Thanks for the pointer. I actually am using sqlite for the metadata, but not the data. I was thinking that sqlite wouldn't be happy with 40+ GB of files, though I haven't tested it. This page seems to say it's doable with some tuning [1]. (Although as mentioned I may go up to 500 GB or more later)

It's nice to see that intuitive performance characteristic actually tested -- small files perform better when stored internally, but large files perform better when stored on the file system.

I want to use rsync to sync files between machines, so that's a consideration. Otherwise I would have to roll my own.

With sqlite you'll also get duplication of the OS buffer cache in user space. Not sure how much of an issue that is in practice.

I like sqlite a lot, but it seems to have more nontrivial data structures than a file system, so it's harder to reason about performance and space overhead. It also has more concurrency issues than a file system.

One thing I like about the file system is that I was able to account for the waste within 1%. I wrote a simple script to compare the waste with 2K ext4 block size vs 4K block size, and it predicted 7 GB of saved space (out of 40 GB or so), and that turned out to be exactly what happened in practice. I don't think I'd be able to account for space overheads so exactly in sqlite (in theory or in practice).

The other dimension is IOPS as mentioned here, but I think that will be easier with just a file system too. There are more observability and debugging tools with file systems.

It's possible that because the inode issue, I could go for storing small files internally and big files externally though. The distribution of sizes is pretty much Zipfian. That isn't too hard to code, and the complexity may be justified.

[1] http://stackoverflow.com/questions/784173/what-are-the-perfo...

Actually I just checked, and the bottom 1.91M files out of 5M are less than 1024 bytes, and sums to only 1 GB ! The other 3.1 M files sum to 34 G (largest file being 2GB or so).

The distribution was more skewed than I thought. So I think the hybrid solution with internal sqlite for small files may be a good idea. Thanks!

ext4 is usually fine, but it has really bad corner case behavior with respect to large directories (> 10k dentries). Iterating through them becomes increasingly slow as they grow larger -- worse than O(n), which is the expected case.

What's worse is that even if you then delete all the files in the directory, performance does not return to normal; iterating becomes just as slow at that point with 10 files. The only solution is to remove the directory and recreate it.

We've been running XFS on our 1000+ hosts for 4 years without issue. There are old stories of XFS issues, but it's been rock solid for quite some time, at least since RHEL 5.

ext4 is comparable to NTFS or HFS+. It mostly works, but other filesystems like btrfs or ZFS give you proper data integrity (checksumming) and other features.

Isn't btrfs still plagued with stability issues/bugs though? (and zfs isn't readily available in default kernels?)

Reading planet.debian.org I seem to get the impression that btrfs it's not 100% there yet, unless you feel a little adventurous.

BtrFS will stop being "unstable" when development is done, in 20 years (much like ext*).

I run BtrFS on my 2 laptops (one with SSD, one with HDD): the only issue I ran into is metadata space exhaustion: I have well over 100 snapshots and suddenly, the drive becomes unwritable. Once you know that can happen, you're more careful about backups and when to remove the old snapshots.

Also, Debian might not be the best place for bleeding-edge stuff like BtrFS: you should consider a distribution with a more recent version of kernel and utils (archlinux, gentoo...)

For a btrfs NAS use case, I suggest Rockstor. It ships with bleeding edge kernels from elrepo and is pretty much exclusively btrfs.

Debian Unstable is usually fairly up to date wrt bleeding edge though :)

ah, didn't think of Debian unstable, my bad.

I might be the minority here, but I've been burned twice with btrfs (in Q3 and Q4 of 2014) and was only able to recover the data with great effort. Both times the errors only occurred after extended use and without a clear cause. My (unfortunately necessarily vague) bug report was met with scepticism whether my hardware was at fault (I ran memtest for 48h without any errors, and never had any problems with other filesystems), which was quite frustrating after I had lost a significant amount of time on their experimental filesystem and at least wanted to help others not have the same problems in the future.

I'll consider btrfs again when a large distribution has adopted it as default and it has worked for them for at least a year.

For what it's worth, SLES 12 was released October 2014 and has btfs by default:

https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12/#intro...

openSUSE also has it default in 13.2 (November 2014):

https://en.opensuse.org/Portal:13.2

Yes, and I still wonder at the sanity of making it the default given the large number of known problems.

I've lost data, irrecoverably, with Btrfs on at least two occasions--the filesystem was completely toasted. More recently I've encountered the unbalancing issue--the filesystem would become read-only approximately every 36 hours with sustained intensive use. That's simply unacceptable--the default filesystem needs to be utterly reliable, and becoming unusable at unpredictable intervals is not suitable for production.

> Reading planet.debian.org I seem to get the impression that btrfs it's not 100% there yet, unless you feel a little adventurous.

Netgear evidently disagrees and has built consumer-grade NASes based on btrfs.

http://www.zdnet.com/article/netgear-revamps-readynas-storag...

I'm also using btrfs for all my home-devices and I've yet to suffer any issues.

I've been running raid-1 btrfs on workstations and a server for a while, haven't seen any issues at all. I did have an issue on a btrfs laptop, it didn't cope with on/off all that well, but that was more than a year ago, maybe I will retest.

I'm a ZFS user but recently I took the plunge and tried btrfs out. I'm mostly impressed. This was after an encouter with ZFS's horrendous performance with dedup enabled. btrfs at least lets you manually dedup (which works fine for my purposes--I rely heavily on hashdeep in my workflows already and it's trival to locate duplicates in hashdeep inventories).

My sense is that it's the more exotic parts of btrfs (i.e. the native RAID but not-RAID-0/1 parts, esp the stuff intended to be in the RAID-z alternates) that have the most issues.

But ext4 doesn't natively support any of that either, so maybe any place you would consider using ext4, btrfs is potentially ok as an alternative. I mean if you approach it as ext4 with a few very useful missing features I think you hit the most useful and widely used corners. You can still use md etc. I'm not talking about datacenter scale things, I'm talking workstations and laptops.

I currently run a zfs-on-linux RAID-1 and have been testing btrfs for one of my backup copies. There are parts of btrfs things seem to work fine and that I really like a lot compared to the ZFS versions. I'm a bit paranoid about data loss and corruption (I do research with large sets of medical images, it's sort of like video except the data is typically very compressible, but few tools work with compressed images--filesystem-level compression is great for this stuff) so I keep hashdeep inventories of everything to document provenance. One thing about medical image processing workflows is you easily end up with multiple copies of images in different directories. I hoped to use ZFS deduplication for this but it is a nightmare. With btrfs you at least have the option of safely deduping files manually using the (cp --reflink) so you can dedup periodically (or even based on knowledge of how the data is layed out in the filesystem) or add it into workflow scripts which works well.

Unfortunately, the one thing that I haven't figured out is how to make a bit-by-bit clone of an existing btrfs filesystem. You can dd, but that leads to issues because device UUIDs are imbedded into the metadata. Working on btrfs, I've managed to consolidate 5.1TB of data that includes very compressible source images, duplacates and a lot of text files into 1.2TB, but I can't figure out how to correctly duplicate the 1.2TB version of the data without it transiently exploding back to 5.1TB in the process.

I do like the way ZFS approches the concept and organization of "datasets" better than btrfs's approach though. btrfs's approach seems more adhoc and less opinionated. I think if you lack discipline and experience with large datasets, btrfs can enable you to do unwise things that will seriously bite you in the butt down the road, whereas ZFS enforces some discipline. I think it's probably because ZFS was designed by people that have seen hell.

> Unfortunately, the one thing that I haven't figured out is how to make a bit-by-bit clone of an existing btrfs filesystem.

Have you tried btrfs send/receive?

No, I have not actually tested it yet, but according to my research the current state of btrfs send/receive is this:

1. btrfs send streams the files (i.e. it decompresseses files that are compressed on disk)

2. The send stream (optionally) is compressed

3. btrfs receive (optionally) recompresses data into the destination

If you have a large dataset and take the time to use one of the slower compression algorithms/settings this means that you have to wait for that compression to happen all over again at the destination. (you could have different compression settings on the two btrfs filesystems, btrfs send/receive is one of the ways for migrating data between these settings)

EXT4 is arguably the most mature but even recently they found a long term bug in EXT4 on RAID that no-one noticed for years

https://www.phoronix.com/scan.php?page=news_item&px=Linux-4-...

Actually it looks like they found a bug in md (RAID) that was causing fs corruption. To my reading it looks like ext4 was just incidental.

Reply on news.ycombinator.com