Hacker News Reader

Comment by FPGAhacker | Hacker News Reader

Disk space isn’t cheap on a Mac laptop.

Yeah, but that's a customer cost, not an Apple cost.

In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."

> In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."

For what it's worth, this can already happen in APFS. While they don't have real deduplication, if you copy a file within Finder, that copy won't take up extra space.

How granular is the COW? If you touch a byte, does just a block get copied or the whole file?

Just a block

It depends on the application. Often applications do a kind of COW on their own, behind the scene they write out a new file with a randomized naming scheme, when that succeeds, do a rename/replace. So in this case you're getting a new file, as in, a file with a new inode.

If your application is smarter, then it could just COW one block, whatever the minimum block size is for that file system. e.g. for Btrfs it's page size, so x86 that's 4KiB, ppc64le it's 64KiB. I'm not sure what it is for APFS, I'd guess 4KiB. It could actually be inefficient depending on the use case, e.g. a 1MiB file with a single byte change in the middle, if it were a single 1MiB extent, becomes three extents with COW:

    0-500KiB - extent 1
    500-504KiB - hole where stale data is
    504KiB-1MiB - extent 3
    4KiB   - extent 2, with new data, a copy of 4095 bytes from the above hole, with 1 bytes of new data.

Edit: 2nd paragraph suggests the application would COW if it's smarter; that's not correct. If it's smarter, it would request an overwrite of some portion of the file and the file system is what would COW (if it's a COW file system). Granularity depends on a combination of application and file system.

Edit 2: More detailed answer on Btrfs. The whole file system on a zram device is about 260 lines using 'btrfs inspect-internal dump-tree' command. I'll excerpt just the file extents.

1. 'dd if=/dev/zero of=/mnt/test/bunchofzeros.txt bs=M count=1' results in this:

    item 7 key (257 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 7 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 0 nr 1048576 ram 1048576
    extent compression 0 (none)

2. Using vi to change one character and save as the same file, ':wq', I get:

    item 7 key (262 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 9 type 1 (regular)
    extent data disk byte 7401472 nr 1052672
    extent data offset 0 nr 1052672 ram 1052672
    extent compression 0 (none)

So you can see it's a new file, new inode, and looks like length is changed too, maybe something to do with encoding.

3. Delete that file and create a new one with the same 'dd' command as in 1.

    item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 11 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 0 nr 1048576 ram 1048576
    extent compression 0 (none)

    [root@flap ~]# dd conv=notrunc if=/dev/urandom bs=1 seek=524290 count=1 of=/mnt/test/bunchofzeros.txt 
    [root@flap ~]# hexdump -C /mnt/test/bunchofzeros.txt | more
    00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    *
    00080000  00 00 5d 00 00 00 00 00  00 00 00 00 00 00 00 00  |..].............|
    00080010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    *
    00100000

    item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 11 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 0 nr 524288 ram 1048576
    extent compression 0 (none)
    item 8 key (266 EXTENT_DATA 524288) itemoff 3356 itemsize 53
    generation 14 type 1 (regular)
    extent data disk byte 5242880 nr 4096
    extent data offset 0 nr 4096 ram 4096
    extent compression 0 (none)
    item 9 key (266 EXTENT_DATA 528384) itemoff 3303 itemsize 53
    generation 11 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 528384 nr 520192 ram 1048576
    extent compression 0 (none)

Clever. item7 and item9 actually reference the same original 1MiB data extent, but use offsets to point to the unchanged portion "creating" a 4KiB hole, and that hole is filed by item 8, a 4KiB extent containing the change. So it really did only COW one block.

Sorry, I can’t tell, are you still talking about APFS?

No the example under Edit 2 is Btrfs. On the details I'd expect APFS to differ. But the gist is that how changes manifest on disk do also depend on the application being used. On any file system, an edit in vi means a whole new file is written out, so it's effectively COW at an application level. Whereas an edit with an application that does byte level overwrites, would get translated into COW on a COW file system like Btrfs, APFS or ZFS. And I'd expect this COW operation to operate at the file system's minimum block level size.

In my opinion storing, not deleting, is the most important job of a file system. Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume.

(If my opinion holds true, then deduplicating is a problem when it comes to storage: a copy made to prevent bitrot/other damage to the original data doesn't work as such.)

Storing vs deleting is one of the reasons that I'd love to see more research done on log based filesystems.

> Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume.

I'm not sure if I agree.

In an ideal world, it makes sense that performance should be paramount. In reality, I think it's often more important that computers do what we expect them to, so we can predict when they'll break and know how to resolve the problem when they do. This is all the more important in consumer software.

"My hard drive is out of space" is a very basic computer problem that most users will run into at some point. The intuitive solution is to delete stuff. If that solution doesn't work, it could be a major problem.

Or even worse: "You no longer have any free space on your hard disk because you made a change to a large file that had a deduplicated copy."

Block level deduplication should limit the impact of this case.

Only if the change is localized to specific blocks.

Besides, block-level dedup is expensive to maintain. Unless you're expecting to find a lot of duplicates -- which probably isn't the case on most end-user devices -- it's a net loss.

Expect about 30%, most of it in smaller system files and libraries. Block-level dedup is great for pretty much anything where you have a consistent workflow.

A consistent workflow sounds like something you’re going to find more regularly on a server than a mobile device or home computer.

It's cheaper than the memory upgrade you need to support dedup properly.

Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).

Then why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?

They do RAM compression, I thought disk compression was soon to follow after Apple went all-flash storage

Presumably not particularly valuable since most of what takes up consumer storage these days (pictures, videos, app assets) is already in some highly compressed format.

APFS does indeed do compression, the rationale being that compute is cheaper than I/O.

Even if most files on a typical personal computer are already inherently compressed (JPEG photos, lossy audio, etc.), you're not losing any performance, and are gaining some extra space where possible.

Relative to what? I've been around long enough it's hard for me to see just about any disk storage as expensive even if it costs more than whatever is on sale at Newegg.

Mac laptops aren’t upgradable without goofy adapters to convert nvme ssd m.2 to apple’s form factor, at which point your case won’t close up without flexing the aluminum or main board or both. For many MacBook Pro users, upgrading storage is buying a new laptop.

That’s expensive in my opinion.

OWC has made drives that fit Apple’s form factor (assuming your laptop is old enough) for a nice premium. Depending on which laptop you have, this can still be a science experiment.

Reply on news.ycombinator.com