Learn Just a Little Awk (2010) | Hacker News Reader

Learn Just a Little Awk (2010)

gregable.com

277 pointsingve69 comments

I don't write much Awk code any more, but at one time it was my favorite programming language, and not just for little one-liners.

A while ago I ran across a couple of my old Awk programs and uploaded them to GitHub. One is an implementation of Mark V. Shaney (the infamous Markov chain program), the other takes source code and prints it out in a nice "two up" landscape format on a LaserJet II, including graphical boxes to replace the ASCII dividers in comments like this:

  //------------------
  // Some comment here
  //------------------

I wish I had a sample printout, but they are long gone.

Awk was the first language I used that had associative arrays built into the language, which made SHANEY.AWK almost trivial to write. Especially compared with implementations I saw in C and other languages of the day.

I think LJPII.AWK is a good example of how you can write a fairly substantial program in Awk and have it be (mostly) readable. (It was written for either MKS Awk or Thompson Awk, I forget which - so there are a few things in it that may not be available in a typical Awk today.)

https://github.com/geary/awk

https://en.wikipedia.org/wiki/Mark_V._Shaney

Awk is a fantastic "swiss army knife", and always included (in some form) on every *nix.

The article mentions FS but this is easier to access with -F. e.g. for some quick and dirty CSV munging-

awk -F, '{print $2}'

Something else I find super useful is the built-in regex matching, e.g.-

awk '/ERROR/{print $(NF-1)}'

or maybe picking out all 503's from an web server access log-

awk '/503/{print}'

edit: formatting....wow hn is bad for this

> awk '/503/{print}'

{print} is the default action, I believe. So it would suffice to do:

  awk '/503/'

Though, then you may as well just use grep for something like this.

Just a basic example, I'd definitely use grep for something as simple as that. Only pointing out the pattern (you'll definitely have some heftier stuff in {} if you're going to bother with awk in that case)

The typical awk case is to print selected fields, or operate on them (count, sum, classify, etc). So a '/<pattern>/ { print }' idiom is fairly common, generally on the way to something more refined.

For some time I've been using both Awk and Perl for text processing, based on which offered the most convenient command for a given task.

There are two (IMO) glaring omissions in Awk:

- using the captured group(s) of a regular expression

- slicing the arguments array

Mawk supports the first functionality, but it's inconvient.

Then at some point, I've found that virtually every task I do in my text processing via Awk can be done via Perl in a comparable way... and much more - including the two functionalities above - so I've ditched awk.

The only thing inconvenient is to type something like `perl -lane` as opposed to `awk`, but this can be aliased :-)

AWK can be learned much faster than Perl though.

Yes, but as soon as your awk script becomes complex, you need to switch to a more complete language like perl (or python or ruby). Switch slightly earlier and avoid learning all the subtelities of awk (the same applies to sed). You need to learn well one scripting language and to know most common use cases of sed, awk.

No you don't. There is no reason to switch. I've written many full-blown programs in AWK and I see no reason to switch them to another language -- they're fast, portable and readable, with tiny memory requirements. It all comes down to mastery: someone who's mastered AWK will have no issues with a large AWK program and therefore will feel no urge to translate that program to another language. Large AWK programs are easy to maintain because the syntax is so small that the code is always intrinsically understandable.

You don't so much learn Perl as develop the ability to emulate Larry Wall.

Not to mention perl is significantly faster. I was fairly bummed when I learned this since it felt ask is no longer that useful.

Try mawk. It's an incredibly fast awk implementation. If you're using gawk, setting LANG=C will make large awk programs with lots of complex regexpes run significantly faster as well.

> for some quick and dirty CSV munging-

Tip: if you have quoted fields, use gawk, and

    FPAT = "([^,]+)|(\"[^\"]+\")";

> The article mentions FS but this is easier to access with -F. e.g. for some quick and dirty CSV munging-

Though if you're doing a lot of csv munging, xsv would probably be a good investment.

TIL! I know a couple other tools exist but this looks pretty nifty. May need to add it to my arsenal

Reminds me of a similar article [1]

Once you get a hang of how shell constructs like pipes, redirections, globs, variable/command substitutions, etc, the various cli text processing tools like grep, sed, awk, perl, sort, tr, pr, paste, etc are worth to learn at least the basics. They have been through years of use and heavily optimized for performance. Just today, a friend of mine called to ask how to improve a Python script's performance for 5-10 MB text file. After confirming he isn't using any other special modules, I advised to check if the performance improves by implementing it using awk.

If anyone's interested in examples based tutorials on cli text processing tools, check out my github repo [2]

[1] https://blog.jpalardy.com/posts/why-learn-awk/

[2] https://github.com/learnbyexample/Command-line-text-processi...

The GNU Project hosts this essay, "The UNIX Shell As a Fourth Generation Language":

https://www.gnu.org/software/nosql/4gl.txt

The concept of language "generations" is mostly lost now, much like how we stopped inventing names for how densely we pack transistors onto silicon once we'd gotten to VLSI (Very Large Scale Integration); the point of the essay is that the shell complete with a userspace is a programming language in itself, and one which is much more programmable and extensible than, say, getting all of your data into a database and using SQL to manipulate it. It's a programmable environment, as opposed to a data silo, where the only tools you have are the ones the silo's implementer deigned to provide.

There is a short talk by Eben Moglen [0] which is only 10 minutes long but jam packed with wisdom. One part which changed the way I think about using computers is how he describes GUI interfaces: as a "point and grunt" or "caveman" interface. Using UNIX, on the other hand, is to use language to communicate with the computer and comes with all the benefits that we have when using it to interact with people.

[0] https://youtu.be/uKxzK9xtSXM

Thinking over this, it just (finally) struck me that awk (and sed, grep, etc.) are basically just libraries (more like DSLs), and the shell is just another scripting language with a REPL/notebook environment.

When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? I don't think ubiquity/portability is a compelling answer, since (for many/most people who care about this) it is relatively easy to install the environment of their choice on their personal computing device. BTW, we must also strive to maintain that flexibility in the next generation of computing devices.

As an aside, folks who do like the task-focused nature of shell DSLs might appreciate language-oriented programming with Racket.

Awk is part of BusyBox, thus it's present by default on a wide range of embedded systems/systems with limited resources.

Some people are comfortable working in sh/sed/awk. I personally don't care for it. I'm not familiar with it because I don't use it very much, and I dislike the lack of consistency and all the gotchas in shell scripting. (I'm talking about sh+sed+awk+other command line tools, not just awk.) But I've done it at times when it's all that's available.

There is no inconsistency if one programs in straight Bourne shell from 1978, not using any additional features, and frankly, I've been programming in it for 25+ years and never needed "bashisms". ksh is also extremely consistent and portable. One of the main hidden strengths of AWK is that there is one and only one way how to do something in AWK syntax, so from that point, as long as one doesn't use GNU AWK-isms, it's programming luxury and paradise. Programs written in non-GNU AWK run everywhere (one would have to go out of one's way to use Thompson specific features, for example). As far as I'm concerned, it's a matter of understanding UNIX, not GNU/Linux, but UNIX, so that it is clear what is what and what specifically to avoid (like any GNU features). People who grew up on GNU/Linux often struggle with understanding what's what because they have a huge knowledge gap. The fact that GNU/Linux was their first UNIXoid OS is actually highly detrimental now.

I suspect shell languages see widespread use because they are optimized for interactivity. The barrier between thinking of an action and making it run is low. At a certain point there is diminishing returns for particularly complex actions, e.g. you call out to awk instead of implementing awk in a shell language all else being equal.

Python and Lisp use way more parens (which many find unergonomic) and don't have pipes. Let me qualify that - parens are cumbersome with the usual tools and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom. That being said, there is a Python shell called xonsh, but that uses kind of a Python - bash hybrid AFAIK.

> and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom.

That isn't quite true. While, yes, a pipe operator would probably be something new that you add to the language, Scheme which Racket originally comes from, actually comes with first-class pipe support.

Scheme makes use of a concept called ports, quite extensively. You can turn files, sockets, string, etc. into ports. Ports are most definitely idiomatic Scheme.

Racket extends ports with this:

    (make-pipe [limit input-name output-name])

Which works to transfer data from one port to another in basically the same way the pipe operator works at the shell level.

To make it work the way people are more used to, you might also combine it with the infix library.

So if you wanted to create a pipe operator for some reason, everything you need is already there, it'd just by syntax sugar.

Heh neat. I guess this is what scsh does.

> When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever?

Well, first:

> possibly better forms of "glue" -- say Python

That's not a better form of glue. First, consider that executables of all languages are standarized to work with text. They get text arguments, text input through stdin, text output through stdout and stderr, text input and output through other files whose paths are given as arguments, etc.

Shell languages are good glue because they work only with text. With the exception of control structures like "if", and "while", all statements in a shell languages are based on text. The command name is just text, the arguments are text. Syntax facilitates the manipulation of their standard inputs and outputs. Command substitution allows one to glue the output of one command as arguments to another. That's what makes shell languages a good glue.

Python, on the other hand, is not as good at gluing these executables written in a myriad of languages because it doesn't put enough importance on this universal interface between languages that is text, nor does it sufficiently facilitate the execution of other executables.

Another thing to consider is that shell languages are meant primarily to be used interactively. Things like putting commas between arguments and parenthesis around them would be a nuisance in interactive use. Having to quote every argument would also be a nuisance. In a not-primarily-interactive language like python, one uses variables more than literals, so it's easier to write variables than quoted strings. In a primarily-interactive language like bash, one typically uses literals more than variables, so it's the variables in bash that require the extra syntax (the sigil, '$'), while literal strings do not (most of the time).

> with a uniform syntax

I'm not sure what a more "uniform" syntax is, but I get your drift that there are warts in typical shell languages. I would also prefer that `if` statements didn't end in `fi` or `case` statements end in `esac`, but I think languages like bash and zsh offer to much to forgo them for such little details. There are more to these languages than these warts that makes them worth it.

You might be interested in the fish shell project. It's a newish shell language with a simplified ("uniform"?) syntax. Personally, I don't use it because it lacks too many features, but it's there if you like it.

> Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer?

That's shell languages. If you meant something like python, you can. Just do `usermod -s $(which python) $USER` to set python to be your shell. I wouldn't last long on it as my shell for the reasons I mentioned, but there's no-one not allowing us to if we wanted.

I guess shell scripting and the standard commands are convenient interfaces to a lot more functionality. The interactive use (and pipes) tend to make the linear processing super terse and easy (but lots of other things become annoying).

In a type-safe language, it’s typically impossible to specify a function which can do as many things as (eg) sort can, and which doesn’t require specifying a huge amount of default arguments on every invocation. Therefore you end up with a limited function, a huge annoying function, or many small functions. In any case, changing from one kind of sort to another (eg how to sort things, what to sort, how to order them, whether to keep unique elements, whether to be stable, whether to take multiple inputs, whether to assume the inputs are sorted (so just merge), and so on) becomes difficult whereas with a tweak-in-many-ways command like sort it is easy. In a language like CL with optional arguments and no types, this problem can be less hard but whereas sort will likely work fone for a large dataset, your favourite library function may not. (In particular it almost definitely won’t use on-disk storage when needed or support parallelism. Partly note that storing objects from your favourite language to disk is probably hard but if you only work on lines of text it’s easy).

The other big difference is the data types. The data type of shell programs is basically a sequence of lines, each of which is sometimes broken up into one or more records by some other separator. In many languages the sequence operations are all about getting the nth element, or extending them, or changing elements, or deleting them, or splitting the sequence up into other sequences. And these operations typically identify the elements by the position. In shell scripts, many of these operations are unavailable or not used. One basically only iterates forwards, processing each element, and one almost never thinks about the position of the element in the sequence. (c)split exists but isn’t commonly used.

Lots of work in a typical language is converting one data type to another, or extracting bits of it, or doing data type-specific things. In shell these don’t really exist (eg you don’t sort as dates, you write your dates like 2019-09-28 and sort them like strings), and so data is simple, and because extraction is flexible (typically a field number or byte positions) lots of the bureaucracy of changing data types is omitted.

These things all often lead to shell scripts being unreliable. But also resilient, flexible and “sufficient”. By “sufficient” I mean that they can do things well enough that the cost for a long term, thorough, or “proper” solution isn’t justified.

> (In particular it almost definitely won’t use on-disk storage when needed or support parallelism...)

The parallelism possibilities of shell pipelines was always obvious to me.

But I hadn't appreciated explicitly the trivial disk storage ability - that is important too. Nowadays, the other way may be important too - that you can create an in-memory filesystem and run scripts that work off files way faster.

Another tool I feel we don't use enough, is 'make'.

Command line tools are very terse, heavily optimized (ie fast), interactive, and easily chainable. I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution.

The other day my boss asked me to find out if there was any second where we processed more than 100 transactions a second on our server in the last 90 days. I needed to go through 100s of GB of data which mawk managed in no time and in a relatively simple one liner. I imagine my idiomatic python script would still be running

Often times, analysis is an evolving thing. As you uncover one answer, new questions arise. It’s far easier imo to tweak my command line pipelines than to modify program flow.

At the end of the day, general purpose programming languages are jacks of all trades and masters of none. They’re inherently held back by their flexibility when compared to DSLs. Text processing is so easy in awk b/c it’s a text processing domain specific language. If you really think that we should use on language for everything, you’re ignoring the value that specialization can provide you, whether in speed, power, or ease of use

> I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution.

Yeah but I would wager very few people actually know awk. I would also wager 90+% of awk programs are pasted directly from StackOverflow.

That's why I like the idea of pawk[1] - it's like awk, except with python syntax instead of some esoteric syntax that nobody knows.

[1] https://github.com/alecthomas/pawk

I guarantee that more people know awk syntax than pawk!

> It’s far easier imo to tweak my command line pipelines than to modify program flow.

Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.

This is true. Some reasons why sh hasn't been replaced by LISP (for example) as the default Unix shell is because:

- it was there first

- grep awk etc are heavily optimized and faster than their equivalents in most languages

- the primary job of a shell is to run other programs, not to be the program. Scripting languages don't generally let you run a program just by typing the name and hitting enter.

All that being said, you can set your login shell to be whatever you want in /etc/passwd, and you're free to login to your REPL of choice instead of sh if that's what you really want.

Isn't lisp older than shell? :)

Yes. I mean it was there first in the Unix environment, so there's tremendous historical inertia that would be almost impossible to overcome.

I think that's the mindset of 'the unix environment'. They see shell + fs as a repl.

Because that's exactly what it was designed to be and does it remarkably well.

I've found the combination of sed & awk to be a really powerful duo.

One of my earliest development projects circa 2002 was to create a complex awk & sed program that took tons of database blobs with inconsistent semi-structured test from a publishing company and converted it to XML, automatically prompting the user for input (using the Pico text editor) whenever there was something that couldn't be resolved. It saved thousands of hours as part of a metadata digitization project that got their journals into ProQuest.

As only a semi-literate programmer at the time, it was significantly easier than trying the same thing with a "traditional" programming language. On occasion I still call to awk withing python code when I need to do a one-off quick & dirty <something>

For those interested, if you can find a copy, there'a truly excellent book on awk:

"The Awk Programming Language", by Alfred Aho, Brian Kernighan, and Peter J. Weinberger

A copy:

http://gen.lib.rus.ec/book/index.php?md5=0AD0779BF3602F1EA22...

Also Arnold Robbins' books Sed and Awk and Effective Awk Programming are excellent, and also detail the many features added since the latest edition of The AWK Programming Language (1988).

http://gen.lib.rus.ec/search.php?&req=awk+robbins&sort=year&...

Buy the books, do not pirate them.

I use a lot of Perl and occasionally some awk. My experience is that Perl is a lot faster (it really shows when you have millions of lines of text to chew).

For example, I compared two scripts that find the longest line in a file: Perl took about a second when awk took 21s.

Same. Also, many tasks grew in complexity until I threw away my awk/sed combos and wrote some Perl instead.

I used to struggle at dtrace until I realized it was just awk for a probes instead of regular expressions.

Prior discussions:

https://news.ycombinator.com/item?id=1738688

https://news.ycombinator.com/item?id=2932450

I wonder. Unless you have this syntax (and their semantics) memorized so that it just flies from your fingers, it does seem much easier to me to use whatever your language of choice is (Python, Java, ...) to whip up a script. Sure, awk will be terser, but in my case at least, the bottleneck will not be typing, it will be looking up awk features and syntax.

Maybe if you do this every day though. I only have need for things like this once every few months.

I love awk, grep and sed.

But I am wondering: in a professional setup, are there no better tools to analyse and interpret structured log messages out there?

I recently learnt awk but had a hard time figuring things out (had been trying to learn it since 6yrs)

So I wrote a small guide which is hands on for awk

https://github.com/thewhitetulip/awk-anti-textbook

Awk and sed can provide a better integration with "modern" formats like JSON/YAML/Protobuf/etc. And with corresponding tools like jq/pq/rq. It would help to keep them popular still.

403 Forbidden

Why is this linking to the wayback machine? This blog post is live on the same website at the same url.

https://gregable.com/2010/09/why-you-should-know-just-little...

Some people link to archive.org to future-proof against dead urls.

That’s kinda backwards. One should point to the source and if that is no longer available then fire up the Machine.

In this case the archive.org link is a 403 for me while the original works.

I like what Firefox does where if a site 404s it checks to see if there is a Wayback Machine snapshot for it and automatically loads it.

Firefox doesn't do that. An extension to Firefox does.

I suspect you misunderstand some of the point.

When you make a posting on HN or anywhere else and cite a URL, that URL could go down years later. Your historic posting then has a dangling URL. You can't go back and edit old postings to fix it.

Also, some content changes. The URL is valid, but the cited material is not there. With the archive, you can point to a specific date when that was retrieved.

I get a 403 as well, no point in linking to something unaccessible just because reasons. Gdpr is hard but people in Europe should not be denied access just for the sake of immutable URIs

Wait what? https://web.archive.org is inaccessible to you because you are European? FWIW: I am European, and it works for me.

it's kind of rude, it's a bit like Google showing large snippets in the web search results so that users never have to leave google.com

Yeah, that's bad. It's important to use the canonical URL, among other reasons so the site name shows up correctly. Changed now.

ingve is an excellent submitter who doesn't normally do that, so I'm sure there was some reason.

I think the traffic is too much. 403 Forbidden.

Weird. The site is served by cloudflare and backed by an s3 bucket. Totally static. You shouldn't get forbidden unless cloudflare or your isp are doing something odd.

Oh nevermind, the original url was archive.org apparently.

A lot of the examples are just reimplenting cut.

I have, and read, a book on awk (the AWK programming language, by guess-the-initials) and didn't see the point. If I need regexes I have emacs. For CSVs I use excel or LibreOffice.

It has been a while so could someone remind me why awk is so fab? Perhaps it's just I've never met the right use case to make me fall in love.

Sorry if it's a dumb question. TIA

AWK's primary use case is processing a file line by line. It offers a few well thought out mechanisms to do so, allowing you to script relatively complex operations in just a few lines. At the same time, and acknowledging it's not pretty, it's powerful enough to run an http server (not kidding: https://github.com/kevin-albert/awkserver).

So if you want/need to write a script that requires more sophistication than cat, join and sort offer, but you don't want the hassle of setting up IO, field splitting, pattern matching in a more complex programming language, etc., AWK is your friend.

Some of examples of my own awk scripts: converting csv to json, counting trigrams, splitting text by words or sentences, getting the one line from a complex and long log file that I needed, pre-processing files before compilation, cross-tabulation, rewriting file paths, converting weird file formats and ad-hoc templating.

excel can't handle millions of lines. Every job (only 2, but still) I've had, I've had a problem at least once where I'm given millions of lines and needed to parse them for something.

That being said, I used python in those instances, but I'm willing to believe I could've done it faster and more easily with awk.

Yes, this was my feeling too, why not just use python and save the effort of learning another language? Perhaps because you can't assume python will be present in a basic *nix install.

Awk predates Excel (and therefore LibreOffice), so the original point was that...there was no GUI spreadsheet application to use instead.

32-bit Excel used to have a limit of around 65 000 rows. 64 bit Excel is just over 1 000 000 rows. Both of these limits are commonly exceeded.

Excel requires all the data to be in memory. Awk processes line by line so requires less memory.

Awk scripts can easily be incorporated in a chain of other commands.

Version comparison is easier with Awk scripts as they are text based not binary (xls) or compressed XML (xlsx) files.

Awk can be combined with sed, uniq, sort and other tools as one-liners or in shell scripts for fast and powerful data analysis or transformation. These tools are available by default on my Unix-like operating systems.

Nowadays though, scripting languages like Perl (inspired by Awk), Python or Ruby might be used instead of Awk.

Awk has been a handy tool for me professionally a number of times when I had large amounts of textual data to analyse. I have used it many times to transform data to pass to Oil and Gas engineers who were unable to handle the raw data in Excel.

For very large CSV files you cant use excel or libre office because they cannot even open the files. Instead you just use bash tools like awk, sed, grep to get what you need. Even for moderately large CSV files these tools are miles faster than GUI applications.

Reply on news.ycombinator.com