Hacker News Reader

Comment by fragmede | Hacker News Reader

> That is a dataset which is already deduplicated via copy-on-write semantics (it was not using ZFS deduplication because you should basically never use ZFS deduplication).

Can someone more experienced with ZFS say why?

View on HN

"Basically never" is an overstatement, but it is true to the point of "Never unless you already know why I said 'basically never'"

It boils down to the fact that ZFS maintains a mapping from hashes to LBNs. This allows write-time deduplication (as opposed to a scrubber that runs periodically and retroactively deduplicates already written blocks). This is somewhat memory intensive though. For smaller ZFS pools you can get away with just having lots of RAM (and with or without dedupe ZFS performs better the more RAM you have). For larger ones, you can add a SSD to act as additional disk cache.

Here's a quick description of that setup:

https://blogs.oracle.com/brendan/entry/test

Note in this example that they were already showing 128GB of RAM for a 17TB pool; the L2ARC was to augment that. In general, ZFS was designed with a much higher RAM/Disk ratio than a workstation typically has.

ZFS is also very far away from the state of the art in online dedup. For instance, http://users.soe.ucsc.edu/~avani/wildani-icde13dedup.pdf has a theoretical dedup regime that needs only 1% of the RAM for 90% of the benefit.

I'm not an expert by any means, but the most cited reasons are that it requires a very big amount of ram, and it depends a lot (obviously) on the type of data.

In addition to the memory requirements, I seem to recall that it works at the block level, as opposed to the file level. So you could have two of the same file, but maybe one copy is written at the start of a block and one is written in the middle of a block. Same file, different blocks, so no deduplication.

I'm pretty sure that ZFS doesn't do block-suballocation, so that scenario is not possible.

Reply on news.ycombinator.com