Using GNU parallel (YouTube tutorials) | Hacker News Reader

Using GNU parallel (YouTube tutorials)

youtube.com

2 pointsuranium2352 comments

We've briefly used parallel for an internal tool when the site went "multisite" (i.e. multiple parallel instances that sometimes need to all do the same thing, such as post-deployment tasks).

We currently have a patch to remove it, and instead use regular task backgrounding and the `wait` shell builtin.

The way output is buffered was not particularly conducive to what we wanted to achieve.

THE way? It sounds as if GNU Parallel only buffers in one way.

GNU Parallel can handle output in a few ways:

  * --ungroup: With -u output GNU Parallel does not touch the output and output may mix
  * --line-buffer: This buffers a single line from each process.
  * --group: This buffers in files in $TMPDIR. $TMPDIR can point a RAM disk, if you want to buffer in RAM.
  * --sqlworker: This buffers in memory before saving output to an SQL server.

If you are using '&' + 'wait', it pretty much has the same effect as `--ungroup`.

Reply on news.ycombinator.com