Hacker News Reader

Comment by rdtsc | Hacker News Reader

Sneaky pro-tip - use Makefiles to parallelize jobs that have nothing to do with building software. Then throw a -j16 or something at it and watch the magic happen.

I was stuck on an old DoD redhat box and it didn't have gnu parallel or other such things and co-worker suggested make. It was available and it did the job nicely.

View on HN

Seconded. Really good for preprocessing of data files, or repetitive data pipelines. Just set up a naming convention for the intermediate files and define transformation rules. Then you can drop one new file in somewhere, type "make", and the minimal set of processing will just run.

If you need more power, use the wildcard expansion and "patsubst" type rules to define rules at runtime.

Gnu xargs since circa 2009 has had a '-P' option that will let you execute simple tasks in parallel.

I forgot why I couldn't use xargs -P there. If I had to guess ancient DoD version of RHEL, like say RHEL 5. Note it was released in 2007 and xargs as you mentioned had that since 2009.

Sometimes Redhat backport things but I just remember not finding anything there and then being very happy to have discovered make. I think even with xargs -P I would have gone with make anyway as it involved a few dependencies and checking of file times.

For more, see https://news.ycombinator.com/item?id=14836340 (Ask HN: In what creative ways are you using Makefiles?).

I use a build system, redo rather than make, to rebuild my GOPHER site and Debian/FreeBSD package repositories. (https://news.ycombinator.com/item?id=14837740 https://news.ycombinator.com/item?id=14928216)

Fun fact: GNU Parallel used to be a wrapper for making Makefiles: https://www.gnu.org/software/parallel/history.html

Also see this if you have a system without GNU Parallel: http://oletange.blogspot.dk/2013/04/why-not-install-gnu-para...

Reply on news.ycombinator.com