Can you elaborate a bit on how the hashing and chunking works?
There's a rolling hash for determining chunk boundaries, and also SHA-512/256 somewhere.
Does the same data chunked differently have a different hash?
Does the same data chunked differently have a different hash?
sha-2 is used to compute the hash of individual chunks. This is the classic use of hashing in content-addressed systems.
We also use a rolling hash to compute chunk boundaries. We do this in the typical way that tools like bup, camlistore, rsync, and others do for large files.
But our observation was that if you squint your eyes, a merkle tree looks a little like a b-tree. So we use a rolling hash to break up huge lists, maps, and sets into trees where nodes are roughly 4KB. So it's a kind of self-balancing, probabilistic, deterministic b-tree thing.
If I start with integers 1-1000000 and you start with integers 0-999999, and we both make mutations to converge at the same list, we will end up with the exact same tree, with the exact same hashes.
This is what makes efficient synchronization and diff of noms data possible.
Does that mean that your chunk sizes are kind of fixed? Do you think there's a way to retain that advantage and be able to coalesce smaller chunks into larger ones?
Say your smallest nodes are 4KB but for more efficient storage you might want to go up to 4MB chunks. Could that be done while retaining the same hash for the same underlying data?
In any case, that's a good question - we might want to do something about that down the line. But, if we did change that constant, the structure of the trees will change, and all[1] the hashes will change.
[1] a small number will stay the same