Hacker News Reader

Comment by known | Hacker News Reader

cat 100GB_data_file_with_DUPLICATE_lines | parallel --pipe awk \'\!a[\$0]++\' > data_file_with_UNIQUE_lines

is BETTER since it uses less computer resources than

gawk '!a[$0]++' 100GB_data_file_with_DUPLICATE_lines > data_file_with_UNIQUE_lines

The parallel example will definitely not work as you expect, and will likely result in most duplicate lines still being present. When using --pipe this way, if you don't declare --block to be the size of the file (in which case there's no benefit to using parallel), each parallel execution will be run on a separate 1MB (default --block size) chunk of the file before outputting results all separately, then together in a single group (stdout), to the output file.

If you're looking to spread work across CPUs and correctly get the desired output, I'd do something like:

    parallel -a input.txt --pipepart mawk \'\!a[\$0]++\' | mawk '!a[$0]++' > output.txt

I used mawk because it is typically much more performant on large files.

That's a classic misuse of cat.

Just use '<'.

Yeah, but to me, this "misuse" actually makes the input and output clearer than using "<" and ">" together. After all, it places input on the left side of the command, and output on the right side.

Except for GNU Parallel --pipe. When you use '<' what you typically mean is '--pipepart -a file'.

Reply on news.ycombinator.com