Hacker News Reader

They're using pigz on [1] so they are compressing?

I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure.

socat > nc BTW, and does multicast.

As for ssh, it's a shame the "no encryption" option was removed.

View on HN

Wow, I completely missed pigz in there for some reason and had to look it up.

I agree with you on all counts though!

re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers.

We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems.

I'll definitely give socat a look though, thanks for the tip.

Oh, OK, I agree, for 2-3 machines reliability isn't an issue, I was thinking more dozens.

Reply on news.ycombinator.com