Hacker News Reader

Comment by jrumbut | Hacker News Reader

The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following:

col1,col2,col3

1,2,3

4,"hello, \"world\"",6

"7 buckets",,9

To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant.

I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great.

I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.

View on HN

There is an answer to CSV mode a bit further down the page

https://news.ycombinator.com/item?id=28708145

...but if your files are CSV, there is a CSV extension for gawk

    @include "csv"
    BEGIN { CSVMODE = 1 }

Well there you go, for the sake of my pride at least it's an extension.

It's funny searches for awk CSV seem to yield a bunch of SO questions where the answers are increasingly cumbersome regexes instead of this extension.

Of course, you can't count of this extension being widely installed, but it's great for my own desktop.

that's because the extension only works in gawk. its not portable anywhere else.

You can just use https://github.com/Nomarian/Awk-Batteries/blob/master/Units/... and use as so

    awk -f ./ucsv.awk -e '{print $5}'

Also this

> 4,"hello, \"world\"",6

Is incorrect per https://tools.ietf.org/html/rfc4180 so you should just fix it with a sed -i 's/\\"/""/g' and then just parse as normal.

https://github.com/Nomarian/Awk-Batteries/wiki/Formats

'miller' and 'xsv' are pretty good tools for wrangling CSV. (And regexp is kind of a terrible tool for it, too many edge cases.)

Yeah, I don't want to have to write a CSV library each time, that's what I'm trying to get at.

I just end up using Python/Perl but I do have a soft spot for awk so it would be cool if good support was built-in.

Who's writing a library? Just use xsv or miller to extract the bits you want from the CSV, change the delimiter or escapes to something more convenient, etc., then feed that to awk or other CSV-unaware text processors.

I was agreeing with your point about regexes, that it's good to avoid trying to deal with all the corner cases yourself when you're just trying to write a small script.

Ah, understood! CSV is funny, it seems like a more trivial thing than it really is, and its human readability sort of invites broken approaches in a way that something like Parquet would not.

XML is somewhere in the middle--I've seen some horrible abuses of CDATA sections way back when--but at least there are accepted ways to prove what's invalid.

There is a small program I wrote called csvquote[1] that can be used to sanitize input to awk so it can rely on delimiter characters (commas) to always mean delimiters. The results from awk then get piped through the same program at the end to restore the commas inside the field values.

In principle:

  cat textfile.csv | csvquote | awk -f myprogram.awk | csvquote -u > output.csv

Also works for other text processing tools like cut, sed, sort, etc.

[1] https://github.com/dbro/csvquote

Reply on news.ycombinator.com