Comment by 1vuio0pswjnm7 | Hacker News Reader

Comment by 1vuio0pswjnm7 | Hacker News Reader

Reading WSJ without Javascript

Examples

    curl -A ""  https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829|hxextract p /dev/stdin > 1.htm
    firefox ./1.htm

    curl -A ""  https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829|grep -o "<p.*</p>" > 1.htm
    firefox ./1.htm

    x=$(echo x|tr x '\34');
    curl -A ""  https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829 \
    |tr -d '\12' \
    |sed "s/<p[^a]/$x&/g" \
    |tr '\34' '\12' \
    |sed -n 's/<\/p>.*/<\/p>/p' > 1.htm
    firefox ./1.htm

    links -no-connect https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829

The hxextract needs a '-' at the end to read from stdin, otherwise curl ends with error 23 indicating pipe closed prematurely. You also need a space after the '-A' or it sees the '-s' as the user agent name. The double quotes adjacent to the '-A' does nothing in that context.

Forgive me, I do not use curl myself, except in HN examples.

As a matter of practice, I never send a User-Agent header. The localhost forward proxy I use does not send it, nor do the HTTP client utilities I write. If some application sends the UA header, the proxy removes it.

How to remove the User-Agent in curl, without editing the source. The idea to use -A "" comes from the curl help text:

If you give an empty argument to -A, --user-agent (""), it will remove the header completely from the request. If you prefer a blank header, you can set it to a single space (" ").

Another way is to use -H:

   curl -H User-Agent: https://example.com

One might think that would send a UA header with no value, however it actually removes the UA header.

One of the reasons I dislike curl is that there are too many options and the program is constantly changing. I would have to check older versions but I seem to recall in some past version(s) leaving off a space between the option and the option value had some (useful) effect in curl. Maybe not. At any rate, I apologise for the mistake.

Thank you for the correction.

Reply on news.ycombinator.com