Curl Cookbook | Hacker News Reader

Curl Cookbook

catonmat.net

184 pointsingve16 comments

A bit out of topic but curl can as well gives you timing details in case you want to know how long each step of a request takes, here is a nice post about it: https://blog.josephscott.org/2011/10/14/timing-details-with-...

I saw a similar curl post last week and was inspired to add this to my ~/.bashrc for basic timing and headers info:

  wwwinfo()
  {
  printf "##\n# PAGE INFO\n##\n"
  curl \
  --silent \
  --output /dev/null \
  --user-agent 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' \
  --write-out \
  "    Remote IP: %{remote_ip}
           Port: %{remote_port}
    Name Lookup: %{time_namelookup}s
   Connect Time: %{time_connect}s
  TLS Handshake: %{time_appconnect}s
     Total Time: %{time_total}s
     Down Speed: %{speed_download}B/s\n" \
  "$1" &&\
  printf "##\n# HEADERS\n##\n"
  curl --head "$1"
  }

(It spoofs Chrome latest + Windows 10 user-agent, but this is optional.)

My favourite curl snippet is probably generating C code to be used in other projects

curl http://example.com --libcurl example.c

Would be nice to see some good non-HTTP uses. The manpage says:

> curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP).

I use it for IMAP, it works great!

I even filed a feature request to support different authorization/authentication (e.g. shared inboxes accessed through a regular user account), and one of the contributor implemented it. Had a great experience :)

I do wonder what the point of using curl for SCP would be instead of using the scp tool itself.

If you write a script and it takes a URL argument to download or upload stuff and you use curl for it, you get transparent support for a buttload of protocols easily, so there's that.

Also, the manpage says:

> curl is powered by libcurl for all transfer-related features.

So, that means you get the same benefit when using libcurl from other programming languages.

FWIW, scp is deprecated (but I believe that's the entire protocol, not just the tool, though the tool is ludicrously slow and awkward).

Where you hear that? Deprecated in favor of what other tool? As far as I know, there's no other tool that lets you specify files with arbitrary server-side code, like the following example:

  scp -T server:'$(ls -t * | head -1)' .

In the OpenSSH 8.0 release notes they state that "The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.", which has been widely reported (so if you just searched Google for "scp deprecated" instead of trying to undermine the statement with nothing but a statement of doubt, it would have come up immediately) as an informal deprecation; afaik the only thing stopping them from formally deprecating it is a lack of a direct replacement (such as giving sftp a few more affordances).

> so if you just searched Google for "scp deprecated" instead of trying to undermine the statement with nothing but a statement of doubt, it would have come up immediately

I had looked it up with precisely those keywords[1] and nothing that seemed like scp was deprecated came up. However, I admit I didn't use Google.

Also, it wasn't a statement of doubt as much as it was a request for information and an expression of frustration in case it was true. Thank you for answering, by the way.

[1] https://duckduckgo.com/?q=scp+deprecated&t=ffab&ia=web

The 'dump response headers only' section made me chuckle. I thought I had been doing things the hard way instinctively, but nope. It's a good old 'curl -D - -o /dev/null http://example.com' Though the author uses -s, which I don't because if I'm just dumping the headers I'm probably looking for errors.

Also the '-d <param> -G' combo to do query strings surprised me. I've always just submitted the full url, maybe with a quick grumble when I forget to single quote it all. I actually don't see much of an advantage here, since you still need to quote the parameters, and it keeps the same format.

There's also --resolve and --connect-to, which have some neat uses.

For example, direct a request to a specific server in a cluster. Or send to localhost SSL without -k, so you can still test certificate verification works ( localhost can be useful, because some dumb firewalls block 'external' non-443 SSL).

or having to work with SNI

Since 7.49.0 -H supports passing in a text file (@filename), so you can cut&paste and edit headers from devtools.

This might actually be off topic but is there a way to curl into telnet for a websocket endpoint?

Reply on news.ycombinator.com