Hacker News Reader

Comment by zevv | Hacker News Reader

Or, good enough for 95% of the use cases, simply use 'xargs' which is part of findutils and installed by default on all linux distributions. See the '-P/--max-procs' option in the man page for more details.

View on HN

Right, but it's those remaining 5% of situations where you want parallel execution on multiple remote hosts that make this cool (xargs itself can be replaced 95% of the time with backticks, for that matter). Combining parallel, pies[1], and shepherd[2] makes for some very cool possibilities, which you can check out with the GNU System Distribution[3], which I've recently migrated all my personal systems to.

[1] http://www.gnu.org.ua/software/pies/

[2] https://www.gnu.org/software/shepherd/

[3] https://www.gnu.org/software/guix/

You can use xargs for parallel remote execution too. In the simplest case:

    list-hosts |
    xargs \
    -P8 \
    -n1 \
    -I% \
    ssh % some-command

Sure, and you can also run an ssh command inside backticks too. There's something of a Turing tar pit here.

With xargs I always have the problem that lines get mangled when you have concurrent output, so you have to work around that. GNU parallel prevents this by some magic.

The "magic" is simply output buffering inside GNU parallel...something you should be aware of if the concurrent programs are producing large amounts of output.

If you find yourself depending on the order of output lines, your are probably better off using something like Python Fabric instead.

Reproducing that behavior with just bash and xargs is too difficult for me.

"inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#B...

Reply on news.ycombinator.com