Hacker News Reader

Also on multi-node failures smaller partitions actually reduce the amount of critical data to re-replicate to keep the probability low and even give you ability to prioritize that.

For example losing two nodes with 100 partitions on each might give you only like 5 partitions that happen to have replicas living on both nodes and that need to be re-replicated as soon as possible from a single still existing replica, but the rest of the replicas for 190 different partitions have 2 existing replicas and can wait til those 5 high-priority ones are re-replicated.

I guess this answers Martin's question - "you can choose between a high probability of losing a small amount of data, and a low probability of losing a large amount of data! Is the latter better?" - no, the latter is worse, much-much worse.

This analysis leaves out a critical factor, which is that the distributed system will start re-replicating each partition once nodes are declared to be lost. Clearly, if this can be completed before any other nodes fail, no data will be lost. This is the reason why you want more partitions-- not less as this essay recommends. With more, smaller partitions, you can re-replicate quicker.

In practice though, it's not uncommon for multiple machines to fail at once: maybe you lose power to a rack, or a network switch dies. In this case your partitioning scheme really does matter if you want to minimize the odds of data loss. Using a smaller number of larger partitions can decrease the odds of data loss (though, at the expense of losing a larger amount of data if/when you do lose all N replicas of a partition).

Using many smaller partitions can indeed speed recovery, and sometimes this is the right tradeoff to make. But, suppose any amount of data loss means you have to take your cluster offline and rebuild your entire dataset from scratch: in this case, you may be better off with a small number of partitions per node.

That's effectively a partition, not data loss. It is good to be rack-aware though for data placement to still be available in the case of rack loss.

Let's Supose each chunk of data is one 3 machines.

If each each machine has 3 chunks you now risk data loss of those 2 machines fail before they offload data. However if each machine holds 3X munger of machine chunks then if any 2 machines fail you lose data instead of 2 specific machines failing. So even though you replicate faster, your risk of some data loss is actually higher.

Basically you risk data loss over 1/5,000 the time but have 5000 * 4999 more ways to fail. The only real upside is the actual data loss is likely to be very small.

There's an obvious tradeoff between number of partitions and overhead though. Too many, and the increased network traffic will impact the usefulness of the cluster, especially if they need to be fetched from multiple racks for a single computation.

You sound well informed, I just wanted to point out that HFDS replication is not a simple problem.

Also, hopefully you have more than one disk per node (assuming physical machines) ;)