Comment by freedomben | Hacker News Reader

Comment by freedomben | Hacker News Reader

Parent Full thread

I personally consider awk, sed, grep, cut, etc. to be "part" of bash. I don't think I ever write a script without invoking those venerable tools at least once. I often have many pipes, such as

  some_command \
    | grep -E 'some.*regex$' \
    | sed -e 's/erase_text//g' \
    | sed -e 's/erase_more//g' \
    | awk '{ print $2 }' \
    | cut -d ':' -f 1

I used to reach for Perl one-liners a lot, and still do sometimes when I need a gross regex that `sed -E` can't handle (See Perl Pie[1]), but Ruby has been making an appearance more and more often for handy one-liners. Roblog has a great post on using Ruby [2].

If the script is complex enough and I'm willing to take a dependency on Ruby, that is usually where I turn. Where that line is does change from time to time. Ruby makes it super trivial to call shell commands and process them easily as well, which makes it easy to integrate into existing scripting [3].

Hopefully that answered your question :-)

[1] http://technosophos.com/2009/05/21/perl-pie-if-you-only-lear...

[2] https://robm.me.uk/ruby/2013/11/20/ruby-enp.html

[3] https://stackoverflow.com/a/2400/2062384

You might already be aware of this, but you can chain multiple sed functions together into one command. eg

    sed -e 's/erase_text//g; s/erase_more//g'

But I appreciate your example there is more for illustrative purposes rather than an actual pipeline you have in production.

These types of pipelines are why bash and the traditional tools are so maligned. Expert knowledge of sed|(g)awk|bash tools is necessary otherwise you end up with companies telling you that you can't use them.

some_command | gawk ' /some.*regex$/ {gsub(erase_text,"");gsub(erase_more,"");split($2,a,":"); print (length(a[1]) ? a[1] : "STRING ERROR")}'

I'll definitely agree that it can get out of hand. If it's a script/tool that will have many eyes on it and people that need to understand it, bash tools probably aren't the best way to go. I usually turn to ruby in those cases. But even for one off commands, I find myself using pipes like that all the time and I'm the only one that will ever see it, so if it's greek it's no problem :-)

> sed -e 's/erase_text//g' \

I like the pattern. Copied it to my notes to try next time I am grepping through logs.

I usually just string a few 'grep -v ignore_this' with pipes but then it ignores the whole line. But I can see how erasing some parts of the line would be very helpful sometimes.

Drive by code review, you can end a line with the pipe (also && and ||) to avoid a backslash.

    some_command |
      grep -E 'some.*regex$' |
      sed -e 's/erase_text//g' |
      sed -e 's/erase_more//g' |
      awk '{ print $2 }' |
      cut -d ':' -f 1

Reply on news.ycombinator.com