42 karma · joined January 9, 2023
We are currently working on supporting Windows to Windows. Linux to Linux has lower priority as rsync already provides all functionality, it's just a bit slower on fast connections. On slow connections, rsync and cdc_rsync perform very similarly as the sync speed is dominated by the network.
Linux to Linux is also an option if there is demand, but currently it's Windows to Linux only.
hash = (hash << 1) + random_table[data[n]];
bool chunk_boundary = (hash & magic_pattern) == 0;
per byte. That's only a few ops and very cache friendly. The random table only has 256 entries, 8 bytes each, so it easily fits into L1.Also, most modern compression tools have an "rsyncable" option that makes the archives play more nicely with rsync.
uint64_t hash = 0;
uint64_t magic_pattern = 0b001000010000100001000...;
for (size_t n = 0; n < data.size(); ++n) {
hash = (hash << 1) + random_table[data[n]];
if ((hash & magic_pattern) == 0) {
SetChunkBoundaryAt(n);
}
}
In practice, there's more bells and whistles, but that's the gist of it. By tweaking the numbers of 1's in magic_pattern you can influence the average chunk size (distance between two boundaries). With every additional 1, your chunk size halves. There's no special handling of compressed file types. You'd probably want to do that at a much higher level, e.g. just check for extensions.Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.