shell + shell tools + piping = a league of its own.

As a big fan and advocate of a low level language (C) and a high level language (Python), as a combo, being a very powerful paradigm, I still can't believe some of the things the, shall we call it, shell piping workflow, let's you accomplish in one line and in a matter of minutes.

Here's what you do.

Say you have a text log file. It's "semi structured" (like most log files) in that you can extract quite a bit of information using things like grep/sed/awk, but still not fully parse it (without a very complex parser; we don't want to get into that). What's the first thing you do with the log file? You view it:

    view logfile

(view is just opening the file in vim in read-only mode, if you don't have it, you can use "vim -R"). You inspect the file and then quit. Try this instead:

    cat logfile | view -

(don't forget the hyphen at the end). Again, inspect and quit (e.g., using :q). You might say, what a round-about/inefficient way to view a logfile. But. Now you can do something in the middle:

    cat logfile | do something | view -

inspect and quit. Or more than one thing

    cat logfile | do something | do something else | view -

and you keep adding the piped commands until you're satisfied with your output. Once you're satisfied, you can dump the output into another text file as a "report" by replacing the last "| view -" with "> reportfile".

    cat logfile | many | processing | commands | later > reportfile

and you can do this multiple times to generate multiple report files of various kinds. And you can concatente some of those report files

    cat file1 file2

or put some columns of some of the report side by side

    paste -sd' ' <(cat file1 | pick column) <(cat file2 | pick column)

The possibilities are endless.

To give one example from my shell history. Some times I have a dozens of pdf files open in my linux desktop (using evince pdf viewer) and I wanna restart the computer but don't want to lose track of which pdf files were open. There may be automated ways of doing this but let's say there aren't any. I start with ps ax:

    ps ax | view -

A long log. I want to pick only lines that have evince:

    ps ax | grep evince | view -

Now I included the 'grep evince' line too, which I don't want:

    ps ax | grep evince | grep -v grep | view -

Good. But I don't care about all the columns of ps log, except for the last one (the one that shows the full path of the file). That's column 6:

    ps ax | grep evince | grep -v grep | awk '{ print $6 }' | view -

Looks good. Generate a report file from this:

    ps ax | grep evince | grep -v grep | awk '{ print $6 }' > openpdfs_YYYYMMDD.txt

Then I close all my pdfs, and reboot. And I don't know grep, awk except for very basic things like what I already did. But I know similar tidbits about many other commands. If I have to perform regex search only I use grep, but for search and replace I use sed. Sometimes cut is more handy for column selection than awk. If I don't know the command but I know what I need to do, I just google it, and 99% of the time, I can find a stackoverflow post where a linux based one-liner is mentioned which is pretty much a drop-in for my piping workflow.

Finally, if I want to traverse through each line of the entry and process one by one, I use the while loop. For example, if instead of dumping the file paths of the pdf I wanted to format it a little bit using directory name and file name, I'll do this:

    ps ax | grep evince | grep -v grep | akw '{ print $6 }' | while read -r dfnam; do echo 'Directory:' $(dirname ${dfnam}) 'File:' $(basename ${dfnam}); done | view -

And again, view or dump.

Try this, and you would realize that the possibilities are endless.