Hacker News Reader

Comment by sneak | Hacker News Reader

It is a real bummer that there are still no content checksums available. One can work around this by doing sums and scrubs in userspace by storing mtime+hash in an xattr and periodically verifying and updating them. There is a utility to do this but I forgot the name.

View on HN

For a company which primarily sells consumer products it makes sense to omit hardware checksums: You just detect errors which the end user might never notice, a bit flip in a picture you never look at again might not be that bad.

For anyone who has ever heard of checksums this makes APFS a no-go of course.

I guess NTFS is also a "no go" for file storage then as well.

What about ext4? Oh wait - no file data checksums - so a "no go".

Actually, it turns out plenty of folks have built plenty of storage on these platforms - including APFS.

In particular, for APFS, apple uses ECC for all hardware, uses fletcher checksums for metadata and I believe does a copy on write approach to help handle sudden power loss events etc. It seems to be working out OK for them given the prices they charge.

Your point that other filesystems get by without checksums stands, but Apple does not use ECC for all hardware.

Also, we are left to wonder if their "reliable" SSDs are as reliable as checksums at the filesystem layer (or as reliable as both). I suspect the answer is yes, for the majority of applications running atop APFS (which is to say, iOS). Probably not for video assets on that Promise MPX bay (i.e. "external" non-Apple storage media).

I do t think zfs can handle memory correction without ecc either, so is no better in this case.

All the next gen, copy on write filesystems have checksums: NTFS+ (ReFS) has it, ext4+ (btrfs) has it, but HPFS++ (APFS) has not. Bummer.

At least optional checksums should be supported, then watchOS does not waste time with it but a user with 10k+ mac workstation who might still uses spinning rust for bulk data storage is SOL.

> NTFS+ (ReFS), ext4+ (btrfs)

That's.. an extremely confusing way to put it. ReFS and btrfs are from-scratch new FSes.

is ext4+ really shipping in volume? I checked RHEL, not the default.

Is NTFS+ the default on Windows 10 / Windows Server 2016 or 2019?

APFS is shipping today in quantity / as default and I believe they back migrated literally millions of devices to it something like a year ago.

I don't understand the user use case with a 10K workstation and no raid (ie, with parity etc) + SSD with internal error correction. So this is a $10K workstation with magnetic disks and no other error correcting option / layer? Why do this?

SLES (Suse) is going all in with btrfs, it is no ZFS but better than the equivalent RH solution of offering an LVM or other blocklevel layer of checksums.

Not to mention all the ..BSD or ZfsOnLinux based storage solutions.

I am not denying that APFS is a good FS, but not even having the option of checksums given its contemporaries was clearly a political and financial decision to benefit Apple and not its users. Pointing to an opaque mix of raid (and indeed, raid10 is not even enough), super-reliable SSDs and the full stack control doesn't make the lack of checksums any better.

I don't really understand what the deal is with ReFS. Is this ever intended to reach normal consumers or are they happy with NTFS?

Just because you can't currently see any problems doesn't mean you're not surrounded by them. You can't see the problems because no one has turned on the lights.

I use removable storage. SD cards, USB flash drives, USB hard disks.

Apple’s hardware choices are irrelevant.

Those other file systems were designed a while ago.

If you're going to design one now, is there any reason to not have a checksuming capability? At least as an option, even though it may be default-off.

I’m thinking of moving back from ZFS to APFS if I had such a workaround. Block- or file-level integrity is now the only thing that keeps me back.

What are the use cases folks have with super high reliability needs?

I'm generally more worried about an SSD falling over entirely - so focus is on duplication (raid with hot spare -> offsite S3 backup).

From a user perspective it's effectively a zero cost feature. Apple should add it in.

OK - so they do a block level checksum (no error correction but it would let you know that something went wrong). Now something goes wrong with checksum. Is this a "zero cost" feature to have every app updated to handle this checksum error. What is gained? If you need reliability do something like SSD or a raid option and a backup.

The apps don't have to deal with it. It's a read error. Presumably something they already deal with. There's no operational difference for the end user between a block that can't be read and a block than can be read but with corrupted data. And yes, storage would ideally be paired with another physical device to provide redundancy and a mechanism to correct the error on the erroring device.

Or just pair it with iCloud and "cloud heal" the corrupted local data. Then mark the sector as bad and enjoy longer lifetime for the device.

I just built an external storage server on a supermicro 2U with 80TB of hdd and 2TB of fast flash running ZFS and connected to my Mac via 10GE. Best of both worlds.

https://github.com/ludios/bitscrub

This is one that does it with CRC32, which may or may not be sufficient for detecting bitrot. It is probably trivial to modify to support a stronger hash.

Reply on news.ycombinator.com