Hacker News Reader

I am sure you can just pipe all this so you don't have to use an intermediate gunzip file.

Just ssh the machine, dump the SQL and load it back into SQLite locally.

I've seen a suggestion several times to compress the data before sending. If remote means in the same data center, there's a good chance compressing the data is just slowing you down. Not many machines can gzip/bzip2/7zip at better than the 1 gigabyte per second you can get from 10 Gbps networks.

rsync will transmit only the delta between the source and destination.

Believe there are command-line flags necessary to make it do that though: --inplace --no-whole-file

--no-whole-file is the standard, except for local paths (on the same system, the delta-transfer algorithm would be a waste of work, as it reads src and dst to find the delta with rolling checksums -- over the network it can save lots of time and bandwidth)

--inplace is independent of that: it specifies writing directly to target files (instead of write to temp file and rename to target after completion)

Reply on news.ycombinator.com