Hacker News Reader

Comment by ashton314 | Hacker News Reader

Netcat:

    $ nc -l 4242 > dest

And then on the sending end:

    $ nc hostname 4242 < file_to_send

This works great when you just need to get a file of any size from one machine to another and you’re both on the same network. Are used this a lot at one of my offices to schlep the files around between a few of the machines we had.

View on HN

This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar.

Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive

piping through tar is my "power move" when I want to steer a new recruit. I just do it casually in front of them... depending on whether they are awed or horrified by that, they'll get more hacky or more formal work.

You can also pipe through gpg if you want encryption along the way!

And through gzip/zstd/etc if you want compression!

And through tac if you want it in reverse!

(...And my axe!)

Note that gpg compresses by default. Which is thy it's so slow. You may wish to use lzop or zstd but also disable gpg's compression.

I usually use ssh (e.g. ssh -L 3333:localhost:3333) and avoid gpg, though I may use lzop depending on link and CPU speed.

that's what the -C option to ssh is for

lzop if you gotta go fast :)

zstd easily outruns a gigabit link on most somewhat-compressible data.

What's an example of the syntax of this? I remember seeing a graybeard do something like this to huck some files to another machine.

Paraphrased from a real example:

  tar cf - files morefiles | ssh user@dest "tar xf -"

If you want to insert, say, compression:

  tar cf - files morefiles | zstd | ssh user@dest "zstd -d | tar xf -"

Nice. This to me is a great testament of the power of the Unix philosophy: small orthogonal tools that operate on streams of text.

and if you are not worried about corruption. You might at least want to run a checksum on both ends and compare them.

Nah, unix nerds don't care about data corruption. That's the responsibility of some other layer, so by definition we're not in trouble if the whole thing blows up by our own negligence.

Checkmate, rsync.

Netcat is tcp is "reliable" and has checksums for each packet, as long as the length is right you can be reasonably assured that the file transferred correctly.

Not that extra checksums are a bad idea...

TCP indeed has checksums! But they are known not to offer ultimate protection. I had to deal with hosts corrupting TCP streams due to bad hardware before. Which was luckily detected by higher level checksums.

Is there a streamlike tool (a la sed, gzip, etc) that applies forward error correction? I've been looking for such a thing but can't find one that scratches the itch.

I've done that many times, often in combination with tar and pv as mentioned in another reply. But I prefer to do the listening on the sending side. Feels more like serving than shoving I guess. Also, I remember needing a -q flag on one of the ends for graceful shutdowns.

Reply on news.ycombinator.com