[1] http://us-east.manta.joyent.com/mark.cavage/public/surge2013...
[1] http://us-east.manta.joyent.com/mark.cavage/public/surge2013...
I guess the reason I ask is much of the "manipulate and check" that I do happens before I get things to where a one liner will work. That could very well be a programmer competency issue on my part though. :-)
It's faster than people think it is. Especially when you add in libraries like pandas, it's fantastic for data analysis.
Of course, by the time you get to using pandas, you have to have everything in memory.
This isn't true of python in general, though. For simpler tasks, you can easily write generators to read from stdin and write to stdout.
I'm not saying that it's better for things like log parsing, but for more complicated ascii formats, I'd far rather use python than awk.
That having been said, people who don't learn and use awk are missing out. It's a fantastic tool.
I've just seen one too many unreadable, 1000-line awk programs to do something that's a dozen lines of python.
In contrast, when you're building a custom pipeline in a high-level language, you're optimizing for simple solutions and are not likely to get better performance unless you hit an edge case where the standard tools do really poorly.
http://vkundeti.blogspot.com/2008/03/tech-algorithmic-detail...
It's usually quicker for me to iterate on building up a complex program using existing command-line tools -- up to a point. After that point, I switch to something like Node or Python.
One reason it's faster is that they're designed to be composable. They're flexible in just the right ways -- record separators, output formats, and key behaviors (like inverting the sense of a filter or whatever) -- to be able to perform a variety of tasks, but not so flexible that you need a lot of boilerplate, as with more general-purpose languages. They defer unrelated tasks (like sorting) to tools designed for that, keeping concerns separate.
Take an awk script that reads whitespace-separated fields as input and transforms that, adding a header at the top and a summary at the end. awk's got a really nice syntax for these common tasks, and at the end you're left with a program where nearly all of the code is part of the specific problem you're trying to solve, not junk around requiring modules, opening files, looping, parsing, and so on.
Yes, sometimes the full power of a programming or scripting language is what you need, and in cases it may execute faster (though you may well be surprised -- the shell utilities are often highly optimised), but if a one-liner, or even a few brief lines can accomplish the task, why bother with the heavier tool?
However, that being said, I do notice there is a disturbing trend of command line warriors trying to do absolutely everything on the command line resulting in spending 10 minutes to construct a perfect one-liner when they could have just wrote a python/perl script in 2 minutes.