HNHacker News
TopNewBestAskShowJobs

ljusten

42 karma · joined January 9, 2023

submissionscomments
ljusten··on CDC File Transfer
The same question was asked here: https://github.com/google/cdc-file-transfer/issues/56
ljusten··on CDC File Transfer
Short answer, we didn't need it. While the code is largely cross-platform, there is some work involved when it gets down to the details.

We are currently working on supporting Windows to Windows. Linux to Linux has lower priority as rsync already provides all functionality, it's just a bit slower on fast connections. On slow connections, rsync and cdc_rsync perform very similarly as the sync speed is dominated by the network.

ljusten··on CDC File Transfer
Note that cdc_rsync runs on Windows and syncs to Linux. rsync is a Linux-only tool where you'd have to jump through some hoops to make it work on Windows.
ljusten··on CDC File Transfer
Just the compiled and baked game, not your sources. You still developed on Windows or wherever, the cloud instance was just used for running the game.
ljusten··on CDC File Transfer
Windows to Windows is being worked on, see https://github.com/google/cdc-file-transfer/compare/main...s....

Linux to Linux is also an option if there is demand, but currently it's Windows to Linux only.

ljusten··on CDC File Transfer
We're currently adding support for Windows to Windows to cdc_rsync. If there is demand, Linux to Linux would also be possible.
ljusten··on CDC File Transfer
Just uncompress the binaries on the Windows machine and run cdc_rsync. The Linux component, cdc_rsync_server, is deployed automatically on first run. It is scp'ed to ~/.cache/cdc-file-transfer/bin. So nothing has to be installed on the Linux machine.
ljusten··on CDC File Transfer
I should also note that we used a fairly fast 100 MB/sec connection to upload the data, so the rsync diffing algorithm running at 50 MB/sec is actually a bottleneck. The difference would be smaller on a slower connection, where the network overhead would dominate the results.
ljusten··on CDC File Transfer
IIUC, rsync computes a relatively expensive Rabin-Karp rolling hash (https://librsync.github.io/rabinkarp_8c_source.html) and performs a hash map lookup for every byte. Hash map lookups might not be very cache friendly for larger data sets. In comparison, cdc_rsync only computes

  hash = (hash << 1) + random_table[data[n]];
  bool chunk_boundary = (hash & magic_pattern) == 0;
per byte. That's only a few ops and very cache friendly. The random table only has 256 entries, 8 bytes each, so it easily fits into L1.
ljusten··on CDC File Transfer
For reference, here's the paper that describes the in-place update algo in rsync: https://www.usenix.org/legacy/events/usenix03/tech/freenix03.... I haven't looked into it more deeply, but I think it's possible to apply the same idea to variable sized chunks.

Also, most modern compression tools have an "rsyncable" option that makes the archives play more nicely with rsync.

ljusten··on CDC File Transfer
This is currently being worked on, see https://github.com/google/cdc-file-transfer/compare/main...s...
ljusten··on CDC File Transfer
That's because most Stadia devs used Windows, but the cloud instances ran on Linux, so devs had to copy their games from Windows to Linux. We're currently adding support for Windows to Windows as well.
ljusten··on CDC File Transfer
Nope, luckily we still partially work from home and use Chrome Remote Desktop. The command prompt was running on my Windows machine, which I accessed remotely, so I could capture it with the extension. That wouldn't have worked if I had been in the office that day.
ljusten··on CDC File Transfer
I got the same when I was running Stadia on a 2.4 GHz wifi network. Once I switched to a wired connection, it worked like a charm. 5 GHz supposedly works as well, but my robot vacuum cleaner needs a 2.4 GHz network facepalm.
ljusten··on CDC File Transfer
In a nutshell, the algorithm computes

  uint64_t hash = 0;
  uint64_t magic_pattern = 0b001000010000100001000...;
  for (size_t n = 0; n < data.size(); ++n) {
    hash = (hash << 1) + random_table[data[n]];
    if ((hash & magic_pattern) == 0) {
      SetChunkBoundaryAt(n);
    }
  }
In practice, there's more bells and whistles, but that's the gist of it. By tweaking the numbers of 1's in magic_pattern you can influence the average chunk size (distance between two boundaries). With every additional 1, your chunk size halves. There's no special handling of compressed file types. You'd probably want to do that at a much higher level, e.g. just check for extensions.
ljusten··on CDC File Transfer
>> Or, how is it that the local manages to be clairvoyant enough to choose the same windows?

Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.

ljusten··on CDC File Transfer
We used an internal Chrome extension to capture the gifs. There also seem to be some externally available GIF capturing tools, like https://chrome.google.com/webstore/detail/chrome-capture-scr... (disclaimer: I haven't tried those). We then used https://ezgif.com/optimize to remove dupe/unwanted frames, tweak timings and compress the gif. The actual content was done in Google Slides.