GNU Parallel
gnu.org
gnu.org
xargs -n1 -P4
Would be at most one arg from the arg list run with 4 jobs. http://stackoverflow.com/questions/28357997/running-programs...
In addition, I often use it to search for sequences by running grep in parallel. For example
$ parallel 'grep {1} -f haystack.txt' :::: many_needles.txt
Where {1} is a single line in many_needles.txt
It may seem to be a trivial difference, but then you can search multiple haystacks at once fairly easily, and this approach scales to hundreds of millions of needles at once. The code for it isn't very difficult either, heck you can just use an in memory SQLite dB to get a searchable, temporary, index and rely on using some of the most tested software in history.
Basically using unique attributes of the data you then divide and conquer on those attributes (e.g. for Radix you make buckets based on the digits).
$ fgrep -f many_needles.txt haystack.txtI admit I try to use -f all the time but your post suddenly made me realise I'd never actually measured the effect. :/
Edit: yes sorry I meant -F
There are two possible reasons why one might want to explicitly use fgrep/`grep -F`: 1) to avoid needing to escape a string that otherwise contains special regex characters and 2) because it permits the search tool to avoid the regexp engine entirely.
(1) is always a valid reason and is actually quite useful because escaping regexes can be a bother. But whether (2) is valid or not depends on whether your search tool is smart enough to recognize a simple literal search and automatically avoid the regex engine. Another layer to this is, of course, whether the regex engine itself is smart enough to handle this case for you automatically. Whether these optimizations are actually applied or not is difficult for a casual user to know. I don't actually know of any tool that doesn't optimize the simplest case (when no special regex features are required and it's just a simple literal search), so it seems to me that one should never use fgrep/`grep -F` for performance reasons alone.
However, if you use the `-f` flag, then you've asked the tool to do multiple string search. Perhaps in this case, the search tool doesn't try as hard to do simple literal optimizations. Indeed, I can actually witness evidence in favor of this guess. The first command takes 15s and the second command takes 10s:
LC_ALL=C grep -c -f queries /tmp/OpenSubtitles2016.raw.en
LC_ALL=C grep -c -F -f queries /tmp/OpenSubtitles2016.raw.en
$ cat queries
Sherlock Holmes
John Watson
Professor Moriarty
Irene Adler
Interestingly, performing a similar test using BSD grep shows no differences in the execution time, which suggests BSD grep isn't doing anything smart even when it knows it has only literals (and I say this because BSD grep is outrageously slow).
As a small plug, ripgrep is four times faster than GNU grep on this test and has no difference whether you pass -F or not.
(This is only scratching the surface of literal optimizations that a search tool can do. For example, a good search tool will search for `foo` when matching the regex `\w+foo\d+` before ever entering the regex engine itself.)
(Russ Cox's excellent writeup is here: https://swtch.com/~rsc/regexp/regexp4.html)
For my taste GNU parallel gets recommended a bit too quickly e.g. on Stackoverflow when the standard tools would do just fine. Your linked SO question is a prime example of that, xargs specific question yet there's a response that dismisses it entirely and suggests parallel when xargs can easily do the task at hand.
One of the two is part of a standard package that all Linux systems, and many non-Linux systems, have installed. The other is a special-purpose tool.
Your bet is irrelevant in the face of widely implemented standards. POSIX will outlive GNU coreutils. I'd bet your system also probably has a package manager that makes it easy to mark parallel as a dependency of your software.
>many non-Linux systems
Linux is pretty much the only system that typically ships GNU coreutils. The only one that comes to mind is Hurd.
For exactly this reason, GNU Parallel has the option --minversion.
So in your script you put:
parallel --minversion 20140722 || exit
if the rest of your script depends on functionality only present from version 20140722.
If the parallel in the $PATH is not GNU Parallel it will fail (and thus exit). If it _is_ GNU Parallel it will fail if the version is < 20140722 and succeed otherwise.
find -print0 | xargs -0 -n1 echo
echo -n "file '#1'\0file '#2'\0" | xargs -0 -n1 -I{} echo "the file is \"{}\", ok?"I guess you could argue this is supposed to be so and man pages are doing their job, since they are documentation and not tutorials, but still.
This is actually true. In some of the early Unix documentation or an interview (I can't remember which), one of the core Unix developers said pretty much this, i.e., that man pages were more like a collection of rough notes to help the developers of the Unix system itself not to forget how the commands should be invoked.
I'm not sure what that comic has to do with anything.
Man page standards have places for "See also" and "Examples" and that is good enough.
Did you read the "Reader's guide" which is presented before the first option in the man page?
we have bropages for that alas they don't see much activity.
LESS=+/EXAMPLE: man parallel
LESS=+/Reader.s man parallel
man parallel_tutorial
I wasn't checking that, I was ranting on the general state of man pages. I checked ls, curl, wget, mv - none of which had EXAMPLE sections, and that's where I wrote the comment. Didn't even have parallel on my system.
I do agree it needs some work to be a little more approachable.
The examples are a pain to find - jump to the end (here comes the smack in the face) and see pages of differences to other tools.
Thanks for input.
(The trick to find the examples: LESS=+/EXAMPLE: man parallel )
Or, at least, I don't think so, because I don't know what they are or how to access them.
Can you be a bit more specific why you believe it is a "brick-wall-in-your-face"? Did you follow the "Reader's guide", which is before even the first option is introduced?
Except that is not entirely true: For instance according to the author the '{= perl expression =}' will probably never be supported.
(Full disclosure: I am the author of GNU Parallel. I fully support building other parallelizing tools, but to avoid user confusion, I would recommend calling them something other than 'parallel' if they are not actually compatible with GNU Parallel).
This project is cool because it has a really low overhead, which is really cool if you want to parallelize tasks that are not really CPU intensive (but mostly useless if the CPU usage of each task is high).
But the main problem with rust-parallel is that it is not compatible with GNU Parallel (and according to the author, it probably never will be 100% compatible). If you use rust-parallel to walk through GNU Parallel's tutorial (man parallel_tutorial) you will see it fails very quickly.
(Full disclosure: I am the author of GNU Parallel. I fully support building other parallelizing tools, but to avoid user confusion, I would recommend calling them something other than 'parallel' if they are not actually compatible with GNU Parallel. History has shown that using the same name will lead to a lot of unnecessary grief: e.g. GNU Parallel vs. Parallel from moreutils).
Anyway, I did and am continuing to develop this with Rust as one of it's biggest points. It's a cool project to showcase Rust to both users and other software developers. I can use it to teach others Rust by example with well documented code, and possibly gain some buzz for this great language.
That the application has low overhead is mainly a side effect of choosing to develop this application in a low level language like Rust, but the speed is indeed my biggest selling point in general, for now at least. I'm continuing to commit further development into mitigating costs as much as possible.
I can also point out at there is a benefit to using this over the GNU Perl application for large scale computational tasks. Tasks where you are processing hundreds of thousands to hundreds of millions of inputs. This will keep memory consumption low and reduce the amount of CPU cycles required to process those inputs. You could save anywhere from a few minutes to an hour.
parallel -j1 'ffmpeg -i {2} -c:v libx264 -tune film -preset veryslow -crf 18 -vf scale={1} -c:a libfdk_aac -vbr 5 conv/{2.}-{1}.mp4' ::: hd480 hd720 hd1080 ::: *.mp4
The syntax difference is small for simple tasks.
is BETTER since it uses less computer resources than
gawk '!a[$0]++' 100GB_data_file_with_DUPLICATE_lines > data_file_with_UNIQUE_lines
If you're looking to spread work across CPUs and correctly get the desired output, I'd do something like:
parallel -a input.txt --pipepart mawk \'\!a[\$0]++\' | mawk '!a[$0]++' > output.txt
Just use '<'.
parallel -j 1000 -a urls.txt {wget --recursive {}|lynx --dump|(other unix shell power tools) > url.txt}
gets you a powerfull web scraper that maxes out your internet connection.
Throw some machine learning on top of it and you can do a lot with that. Can't tell you what exactly I did with the ML that being trade secret about to get a patent, but the rest doesn't involve the parallel anyway.
find . -name '*.html' | parallel gzip --best
Generally, using it in places where you would normally use xargs seems uncomplicated.
Spoiler: You can have GNU Parallel kick the machines if the fail, and rerun the job on another machine.