One fun thing is that xargs supports parallelization.
-P max-procs
Run up to max-procs processes at a time; the default is 1.
If max-procs is 0, xargs will run as many processes as
possible at a time. Use the -n option with -P; otherwise
chances are that only one exec will be done.
-n max-args
Use at most max-args arguments per command line.
Fewer than max-args arguments will be used if the size
(see the -s option) is exceeded, unless the -x option is
given, in which case xargs will exit.
So here is a "map/reduce" job which takes the log files in a directory
and processes them in parallel on eight CPUs, then combines the results. find . -name "*.log" | sort | xargs -n 1 -P 8 agg_stats.py | sort | merge_periods.py