Hacker News Reader

Comment by miles | Hacker News Reader

To ensure there are no nefarious redirects in such a massive list, this one-liner comes in handy[1]:

    grep -v '#' steves_hosts | awk '{print $1}'| sort | uniq -c

[1] https://news.ycombinator.com/item?id=11456562

View on HN

    find . -name hosts -exec grep -v '^#' {} \; | awk '{print $1}'| sort | uniq -c
    43029
     204 #
    4289934 0.0.0.0
    10763 127.0.0.1
      18 255.255.255.255
      54 ::1
       1 analytics.shein.co.uk
       1 analytics.shein.com
       1 auxilium.ftb.team
       1 bstats.org
       1 fe00::0
      17 fe80::1%lo0
      33 ff00::0
      17 ff02::1
      17 ff02::2
      17 ff02::3
       1 mcmc.dev
       1 mcstats.org
       1 metrics.shmeeb.net
       1 openeye.openmods.info

And looking at those last ones, looks like some files have a different format. For example, the data/minecraft-hosts/hosts file,

     ~/D/P/hosts   master   grep -irn 'openeye.openmods.info' .
    ./alternates/fakenews/hosts:174307:0.0.0.0 openeye.openmods.info
    ...
    ./hosts:174306:0.0.0.0 openeye.openmods.info
    ./data/minecraft-hosts/hosts:9:openeye.openmods.info
     ~/D/P/hosts   master   grep -irn 'metrics.shmeeb.net' .
    ...
    ./data/minecraft-hosts/hosts:8:metrics.shmeeb.net

Someone explicitly called out and removed the Useless Use of cat but yet left the Useless Uses of grep, sort, and uniq in, I see. (-:

* https://porkmail.org/era/unix/award#grep

    awk '!/#/ { seen[$1]++; } END { for (k in seen) { print seen[k],k; } }' steves_hosts

I'd rather have the readability of the first command than the "efficiency" of your example.

Guilty as charged!

I love awk as much as the next person and I've written some pretty large scripts with it, but the original grep/sort/uniq is a lot easier to understand in two seconds. I don't think it's "useless" at all.

Also your awk isn't exactly identical as it's not sorted, but in this case that probably doesn't matter.

Same, I think I use grep | sort | uniq almost on the daily.

One of the advantage of the array[item]++ / for (item in array) construct is that it avoids possibly expensive sorts (at the cost of generating potentially large arrays), and tends to be quite fast.

That's even if you end up sorting the output by frequency, as the summary listing is typically shorter than the overall input data:

  {a[$1]++}

  END {for (i in array) printf("%6i  %s\n", a[i], i) | "sort -k1nr | cat -n";}

The accumulator/loop idiom is ... fairly readily recognisable to someone familiar with awk.

What is a nefarious redirect? Does it look for entries without 127.0.0.1?

A redirect to some unexpected IP address. Here's what the output looks like:

    190625 0.0.0.0
      3 127.0.0.1
      1 255.255.255.255
      3 ::1
      1 fe80::1%lo0
      2 ff00::0
      1 ff02::1
      1 ff02::2
      1 ff02::3

There's a concern that a malicious list could point domain names to a malicious IP address. I don't think its a big concern with https:// since the cert will be invalid, but it's still a concern.

I suspect it’s very likely that somewhere in the world is a domain-validation server, used by a trusted CA, which has this very anti-advertising hosts file installed onto it.

Why? A CA would be unable to issue certificates for advertising sites with that configuration.

It could be a lesser-known CA, perhaps the national CA of a small country (under 2m people) that normally only issues less than tens-of-thousands of certs and only exists for regulatory reasons (e.g. the country requires all of its own gov services to use its internal CA, while all commercial/popular services use CA based in another country, usually LetsEncrypt)

Reply on news.ycombinator.com