Hacker News Reader

Comment by unilynx | Hacker News Reader

This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar.

Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive

View on HN

piping through tar is my "power move" when I want to steer a new recruit. I just do it casually in front of them... depending on whether they are awed or horrified by that, they'll get more hacky or more formal work.

You can also pipe through gpg if you want encryption along the way!

And through gzip/zstd/etc if you want compression!

that's what the -C option to ssh is for

lzop if you gotta go fast :)

zstd easily outruns a gigabit link on most somewhat-compressible data.

And through tac if you want it in reverse!

(...And my axe!)

Note that gpg compresses by default. Which is thy it's so slow. You may wish to use lzop or zstd but also disable gpg's compression.

I usually use ssh (e.g. ssh -L 3333:localhost:3333) and avoid gpg, though I may use lzop depending on link and CPU speed.

What's an example of the syntax of this? I remember seeing a graybeard do something like this to huck some files to another machine.

Paraphrased from a real example:

  tar cf - files morefiles | ssh user@dest "tar xf -"

If you want to insert, say, compression:

  tar cf - files morefiles | zstd | ssh user@dest "zstd -d | tar xf -"

Nice. This to me is a great testament of the power of the Unix philosophy: small orthogonal tools that operate on streams of text.

Reply on news.ycombinator.com