Ext4 data corruption in 6.1 stable | Hacker News Reader

Ext4 data corruption in 6.1 stable

lore.kernel.org

70 pointsturrini27 comments

I’ve defaulted to XFS since it became available for Linux, and never had any problems. Only once, some time around 2009, did I try ext3 and it forgot to flush a lot of data that were lost. I don’t see why anyone would use ext over XFS.

XFS is all but unmaintained, which may be a good reason to be cautious. Very stable code, but the world may change out from under it.

In a sane system, the world would not "change out from under it".

Linux is a special category of mess.

Why would the world not change?

When XFS was designed, the storage ecosystem was spinning rust attached using PATA and SCSI. We've developed quite a bit since then, with things like SAS, NVMe, and SSDs. Storage have gotten over an order of magnitude more throughput, over an order of magnitude less latency, and several orders of magnitude more IOPS.

Running a filesystem designed for ancient hardware means you won't be taking advantage of several decades of innovation.

The world here means the APIs XFS is built on top of.

Yes, it is possible to maintain stable APIs and even offer multiple versions of. Other systems (even AmigaOS in the 80s) manage to do so.

Linux is a special case.

In what ways would you say that ext4 takes advantage of the latest several decades of innovation, where XFS does not? Both XFS and ext2 were first released around 1993, and it is safe to say that XFS was more advanced than ext2 at that time, given that it had journaling and ext2 did not.

If I wanted something actually novel, I would go for ZFS or one of its clones.

XFS is adding pretty radical features and reworking its internals in interesting ways, it's Ext4 the one that doesn't really see that many changes these days.

I guess this falls into the class of Linux bugs that causes regressions on the LTS kernel, where you'd be safer running stable.

When the Linux kernel is huge (MLoCs!), complex and lacks stable internal APIs, this should not surprise anyone.

There are better ways to do it, and it is being done.

Refer to the microkernel multiserver architecture, with efforts such as Genode (on seL4) or Fuchsia.

How on earth does running a file system in userspace suddenly makes file systems safe?

Also, how does putting N millions of LoC in N projects solve problems magically?

Any of the people who were freaking out over a rare ZFS corruption care to comment?

What about it?

Is this triggered under certain workloads, or has it been chewing data for everyone, and how much?

> Is this triggered under certain workloads, or has it been chewing data for everyone, and how much?

Unless you are checksumming your files through a daemon run on a schedule (IDS style) AND manually review the results every now and then, you can't know.

If you have versioned backups, you can diff against update_day-1 without an additional tool.

I have versioned backups for a few months. Most people don't and can't (it requires a lot of storage)

Even if they have them, the diffing needs to be done.

Given the past problems with ZFS, I think I'll write a summarizer, that creates a database of metadata for each backup: it will be small to keep around, and may be use more easily to determine which backup contains the uncorrupted file.

I use borgbackup so the storage requirements are fairly low if the data does not change too much (which is my case). Also diffs are just a « borg diff » away.

It appears to require O_DIRECT, which is notoriously buggy and no sane program should ever use.

Could you share your source on how/when/if O_DIRECT is required? Have a link?

It's in this HN submission's link. It could be that O_SYNC is also needed.

I had already updated to the problematic kernel, so I resorted to installing 6.5 from backports. Hope this helps.

so everything before 6.1 is ok, and after 6.5 is also ok?

6.1.64+ affected, per reply on thread.

Should be fixed in 6.1.66.

iomap: update ki_pos a little later in iomap_dio_complete

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...

Is what you've linked the patch that should be backported to affected kernels?

As the potentially problematic patch causing the corruption, I found https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... but it's not present in the 6.2.2 I run.

I'm very conservative in the versions or critical components like kernels or dkms modules, that didn't protect me against the ZFS bug, but it seems to have avoided me this EXT4 bug!

>The commit got merged in 6.5-rc1 so all stable kernels that have 91562895f803 ("ext4: properly sync file size update after O_SYNC direct IO") before 6.5 are corrupting data - I've noticed at least 6.1 is still carrying the problematic commit. Greg, please take out the commit from all stable kernels before 6.5 as soon as possible, we'll figure out proper backport once user data are not being corrupted anymore. Thanks!

The way I read https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh... 6.5+ should be okay. PS: Please correct me if not!

Reply on news.ycombinator.com