Hacker News Reader

Hacker News

Comment by indescions_2017 | Hacker News Reader

Parent Full thread

indescions_2017

>>> I use sed & awk all the time.

Me too. For example, number of unique IP requests in a log file in milliseconds ;)

$ awk '{ print $1 } ' caddy_log | sort | uniq | wc -l

But rarely compose my own from scratch. It's mostly copy paste. And store in admin bin for future use.

View on HN

No need so many commands :)

    $ cat duplicates.txt
    abc  7   4
    food toy ****
    abc  7   4
    test toy 123
    good toy ****

    $ awk '!seen[$2]++' duplicates.txt
    abc  7   4
    food toy ****

    $ awk '!seen[$2]++{cnt++} END{print +cnt}' duplicates.txt
    2

On a large file with many duplicates, seen[x]++ can overflow, unless you're using GNU Awk with bignums (gawk -M).

that's a good point

I'll add a note, thanks :)

Dumb side note: I always end up using `awk '{print $1}'` instead of `cut -f1` because cut's field separation defaults are unwieldy in that it doesn't intelligently consider any whitespace to be a field separator, which is what I need 99% of the time.

sometimes (if there are not mixed spaces and tabs) tr -s ' ' < input | cut -f1 helps

If you just want to count unique IPs, you can get that in awk alone and avoid sort:

    awk '{ip[$1]=1}; END{ print length(ip) }'

Use the hash tables directly, assuming you've got the memory. That should be slightly faster as it avoids the sort.

nice one

also, {ip[$1]} is enough...

Thanks, I was trying to think of the fastest possible assignment.

There's `sort -u` which should be able to avoid some work.

No need to spawn another process(uniq), you can use sort -u

;-)

For me, the benefit of a trailing uniq instead of a sort -u is that I'm invariably building towards a "sort | uniq -c | sort -n" anyways.

Reply on news.ycombinator.com