Hacker News Reader

Comment by bcyn | Hacker News Reader

So, I don't use awk right now. This answers "how to awk" more than "why use awk" for me (understandable given the 20 min claim).

Does anyone have concrete, practical examples of use cases where awk made your life much easier?

View on HN

I use awk when I need something a bit more powerful than "grep"/"cut", but don't want to pull out the big guns with Perl.

For example, recently I needed to print out certain fields of an output, but only for a given subset. So I used Awk to create a simple state machine (enable when I see the start of the subset, disable at the end), and print the fields of interest.

The grep/cut replacement is exactly what I use awk the most for. I use it for other things too but this stands out. awk just fixes the problem when you need to have more than one field from the input and maybe a different string between them. I guess this can be done with cut too, but for me it's simpler this way.

Also I use awk often in scripts together with fzf to interactively switch contexts e.g. in cloud provider CLIs.

Or you could use it to solve the Towers of Hanoi game ;)

https://rosettacode.org/wiki/Towers_of_Hanoi#AWK

Awk '{ print $2 }' does LWSP gobbling.. cut -d' ' -f2 doesn't and this difference alone makes awk useful to me on a daily basis.

Awk has a hash like perl which is very efficient. I use an awk expression to print uniq as they come in counted through the hash insert on new instead of uniq which prints at end.

Awk count unique over 300,000,000 ips was as fast as perl and python and smaller memory footprint

LWSP=Linear White Space "Linear white space is: any number of spaces or horizontal tabs, and also newline (CRLF) if it is followed by at least one space or horizontal tab." [1]

I don't know what LWSP gobbling is, though. Google didn't help.

[1] https://stackoverflow.com/questions/21072713/what-exactly-is...

It means a contiguous collection of arbitrary white space characters are considered to be a single delimiter.

With the parent example it means that column #2 could have any of number whitespaces between it and column #1.

yes. whereas cut, considers ' ' (two spaces) as an instance of a blank column separated by the -d' ' space instance. Its like ,, in CSV being a blank column.

> Awk '{ print $2 }' does LWSP gobbling

That's like answering "why use Haskell?" with "Because its monads are monoids in the category of endofunctors"

This made me very happy, since it goes to the 'you can either use awk or explain awk but not both' problem which is definitional in monads in Haskell. I thought LWSP was a thing. I forgot not everything is a thing. And gobbling is maybe very jargon ish. Python sys.stdin.rstrip().split() comes to mind.

For the simplest uses, cut is more readable.

If argument expansion is required, consider using read. That way the argument is given a name. Use "read a b c < x" instead of "b=$(cat x | awk '{ print $2 }')".

Just remember to pipe to read with care, as the right hand part of a pipe is a subshell in which variables are local. So "read a b c < <(echo 1 2 3)" works, "echo 1 2 3 | read a b c" doesn't.

Another way to expand arguments is to simply define a shell function and use $2. Something like "process_line() { echo $2 }".

When you have to reach for something like awk, your script would probably improve by being mostly awk. In which case most people are probably looking at perl or python anyway. Even if awk is a nice language, the arrival of perl mostly killed it. It is not wrong to say perl was the next version of awk.

tr will handle the whitespace for you and is generally faster[0] (but awk is faster to type and reads better).

[0] https://datagubbe.se/cutvawk/

If you just use awk for `{ print $2 }` then I would still prefer `tr -s ' ' | cut -d ' ' -f2`, since both are part of coreutils and with `awk` you would add a additional dependency to your script.

that is not equivalent because awk will remove trailing/leading space/tab/newlines (newlines come into play with a different record separator)

whereas, tr will still leave a trailing/leading space

for example:

    $ echo '   a    b  c   ' | tr -s ' ' | cut -d ' ' -f2
    a
    $ echo '   a    b  c   ' | awk '{print $2}'
    b

and by default awk splits on space/tab/newlines, whereas in cut example above, you get only space as delimiter

cut has its uses and will be faster than awk, but it depends on the problem being solved

You are right it depends on the specific use case.

It possible to use `xargs -L 1` to trim the separators, but then you would also add `findutils` as deps.

I just wanted to point out that keeping the dependencies of scripts in mind when programming is also important.

IMO, it would be great if 'cut' could accept more than just a single character for delimiters.

e.g. a regex or even a shell-style pattern match would be very useful. `cut -d ' +' -f2` would get rid of the need for 'tr' in your example.

Even just allowing >1 char delimiters would be helpful. A text list like 'foo, bar, foobar' is too much for cut, but it would be fine if cut accepted parameters like `cut -d ', '`

Why would being part of coreutils matter? Awk is part of POSIX, just like tr and cut. Even in very constraint environments you can count on it via busybox.

A perpetual argument is use of too many pipe separated distinct commands. The argument suggests it's lazy to use sed | awk | grep type pipe runs because in all probability sed or awk alone could have done it and you incurred two excess fork/exec() and therefore consumed kernel and userspace beyond your need. The usual rejoinder is "get nicked"

A lot of stuff is part in POSIX, but not all of that is available on every system. Also busybox configurations can vary a lot.

I wrote my answer to this a few years ago:

https://www.bignerdranch.com/blog/a-crash-course-in-awk/

A tl;dr for the following tl;dr: it's great for quick-and-dirty processing of logs.

The tl;dr is that AWK is simple language that lets you use a line-oriented event driven programming model to process text. This abstraction is simple enough to be readable and maintainable, but powerful enough to parse and process text with line-oriented structure.

The example linked elsewhere is worth studying, because it truly is one of the prettiest pieces of programming I've seen in a long while:

https://c2.com/doc/expense/

If you don't know anything about AWK, here's all you need to know to grok this:

1. An AWK program is a list of [event] { code } pairs. All pairs are run in sequence on each line of input; if the "event" evaluates to true or is a matching regex, the matching code runs.

2. { code } on its own runs unconditionally.

3. The input is automatically whitespace delimited into fields; $1 refers to field 1, $2 to field 2, etc

4. NF is a special value that yields the number of fields

5. Assigning to fields replaces the text in those fields.

6. ($1+0) != 0 implicitly converts $1 to an integer; if it fails, the value is 0.

There is a lot of implicit loose typing in the language, and usage of undefined variables is idiomatic. Functions aren't easy to use, either. So it's not well-suited for programming in the large, or even the medium. But for the scale of programming seen in that link, it's truly a wonderful and simple power tool.

I use awk to write csv-parsers for my personal finance setup. They take transaction logs and convert them into a ledger format (https://www.ledger-cli.org/). There's some pretty trivial if/else branches based on regexp's, which awk is pretty good at expressing.

I'd normally write this sort of thing in python. The awk program is usually smaller than a comparable python program, but for me the main selling point is that awk programs a more "UNIX-pipe-native" than python programs are.

At some point, I'll probably rewrite these programs in a more "serious" programming language, once the file sizes get too big or something, but for now they're working great and are easy to extend.

I've used it for this too, but things tend to go south pretty quick if you have escaped or quoted commas. That, unfortunately, is where I usually break down and pull out python.

hehe, I hit that bump in the road too, but maybe I was too stubborn.

FWIW, you can get around it. I just took a peek because I couldn't remember how to do it off the top of my head.

It looks like I had to use

  gawk -F ',' -v FPAT='([^,]+)|("[^"/]+")'

to get the behavior you're looking for. Seems like I nabbed it from https://www.gnu.org/software/gawk/manual/html_node/Splitting...

Agreed that this is the sane point at which to pull out python.

I hadn't run across that one yet. Thank you!

The risk of that seems to be if CSV allows embedded escaped quotes in a quoted string. Does it? I don't know. And CSV is pretty loosely defined. For most people it's probably "whatever Excel emits or ingests".

And I think that's why we're on the same page about pulling out python and using a module where somebody has explored what the corner cases are and dealt with them for us already.

Yea CSV is famously bespoke, I don’t know if it’s ever been standardized?

In my experience, each implementation is more or less unique. Even if you allow escaping, different formats permit different methods of escaping and you might have to account for each! shudders

It’s much easier to support that in python than it is in awk with regexp. The awk route will eventually make you a regexp wizard, though, which confers it’s own benefits. :)

Here's some articles

* https://blog.jpalardy.com/posts/why-learn-awk/ (also discussed on HN: https://news.ycombinator.com/item?id=22108680)

* https://adamdrake.com/command-line-tools-can-be-235x-faster-...

I contribute to a niche game server project and used awk to generate c# class files from an org mode document that I wrote to specify them.

What about awk makes that easier than, say, a Python script?

Awk automatically operates on every line of a text file, so you can easily use it as part of a pipeline in Linux. I'll usually use grep, cut, sort, and awk all together with one line of code and no need to open an editor. Python takes a lot more code to do these basic things. However, Python is much better once the complexity grows past a certain point.

depends upon the usecase, for relatively simple column filtering, substitution, etc, awk script would be briefer than python equivalent and most likely be faster as well

if it becomes lengthy (and again depends on features required), then Python is likely to have inbuilt/3rd-party libraries to make it easier to write and maintain

if it is a question of constructing command line one-liners and using it as a part of other cli tools, then awk wins easily

for example:

    awk -F'\\W+' -v OFS=, '{print $NF, $2}' input.txt

prints last column and second column, where non-word characters form the field separator and comma is used as output field separator

Processing text files line by line is trivial in awk. Maybe it's easy in Python too but at least in awk you don't have to worry about the 2 vs 3 silliness.

Plus I think semantic indentation is not sane language design.

Plus you can pipe it to another command with ease.

Reply on news.ycombinator.com