Hacker News Reader

Comment by yetihehe | Hacker News Reader

My favourite magic trick is dumping db on one host, gzipping on the fly, tunneling through ssh to other host, then ungzipping straight into other database. Useful when you don't have enough place on disk for local copies.

View on HN

Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120 Mbytes/sec, which is the network bandwidth of a switched gigabit network. There’s no point in making your task take longer with unnecessary compression steps. This also applies to encryption but that’s a much more context dependent decision.

Just about any machine can gzip at > 120MB/s, just use lower compression settings. You can compress even faster with e.g. zstd, as well.

That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either.

    $ lscpu | grep name
    Model name:            Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz

    $ time dd if=/dev/zero of=/dev/null bs=10240 count=102400
    102400+0 records in
    102400+0 records out
    1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s

    real    0m0.147s
    user    0m0.036s
    sys     0m0.111s

    $ time dd if=/dev/zero bs=10240 count=102400  | gzip > /dev/null
    102400+0 records in
    102400+0 records out
    1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s

    real    0m5.364s
    user    0m5.317s
    sys     0m0.472s

    $ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null
    102400+0 records in
    102400+0 records out
    1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s

    real    0m27.352s
    user    0m27.169s
    sys     0m10.119s

    $ time dd if=biggestwords bs=10240 count=102400  | gzip > /dev/null
    10642+1 records in
    10642+1 records out
    108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s

    real    0m6.669s
    user    0m6.667s
    sys     0m0.249s

    $ time dd if=biggestwords  bs=10240 count=102400 | gzip -1 > /dev/null
    10642+1 records in
    10642+1 records out
    108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s

    real    0m1.540s
    user    0m1.522s
    sys     0m0.138s


    $ time dd if=biggestwords of=/dev/null bs=10240 count=102400
    10642+1 records in
    10642+1 records out
    108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s

    real    0m0.043s
    user    0m0.007s
    sys     0m0.036s

Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data.

If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.

It would be super useful to me to see this with a bit more detail, in case you have/know a writeup somewhere!

I didn't invent it myself, you can see example at [0] or googling 'pg_dump ssh tunnel'.

[0] https://coderwall.com/p/bmv0nw/dump-postgres-databases-throu...

For copying a directory tree from host1 to host2 preserving user / permissions and compressing on the fly to reduce network usage i use:

# from host1:

cd $DIRECTORY && tar -jcf - . | ssh host2 "cd DESTINATION_DIR && tar jxvf - "

Yeah this is part of my workflow too.

Reply on news.ycombinator.com