Curl exercises

This is more SysAdmin related, but one power-curl function I use atleast 30 times a day is this alias I have in my .bash_aliases

This will output the HTTP status code for a given URL.

     alias hstat="curl -o /dev/null --silent --head --write-out '%{http_code}\n'" $1  

Example:

  $ hstat google.com
  301

I also use curl as an 'uptime monitor' by adding onto that code section. (a file with a list of URLs, and "if http status code !=200" then email me.)

There are variations on this all over the place but I really depend on it and I like it.

I really like using HTTPie (https://httpie.org). Much friendlier syntax then curl, formats the output...

Also works great with fx (https://github.com/antonmedv/fx). Just pipe the output from HTTPie and you get easy json processing.

Last year, I submitted a patch to Mozilla Firefox which fixed some major issues with the "Copy as cURL" functionality: https://bugzilla.mozilla.org/show_bug.cgi?id=1452442 (namely, that it would stop working in a number of common cases, e.g. after opening the Response tab, or while the request was in-flight). That particular experience taught me a great deal about Firefox and cURL options, plus I had the opportunity to fix a tool I use very frequently.

cURL really is an excellent tool for web debugging. Using something like "Copy as cURL" makes it possible to generate a request which is textually identical to a request made by a real browser, which is really important when debugging subtle weirdness in a browser (or, exploring a web server for bugs!). It's easy to script and comes with a decent progress UI. It's also practically omnipresent and slightly more common than wget - I often need to use foreign systems where I have to make do with the least-common denominator tools, so learning the ins and outs of cURL have been quite handy for me.

This one is handy, if you want to create your payloads dynamically:

    cat payload.json | curl $url --data -@-

You can use this to re-play access logs, like so:

    sed -n 23p accesslog.json | jq '.args' | curl $endpoint --data -@-

Curl is really handy if you want to write a quick crawler for some globs of urls.

For example if you want to download a gallery of images with some fixed format you can do

  curl "http://example.com/imgs/[01-99].jpg" -o "#1.jpg"

And it will fetch them and name them with the correct zero padding.

Or if you want to download the first 10 pages of hn:

  curl https://news.ycombinator.com/news?p=[1-10] -o "hn_page_#1.html"

It can glob pretty complicated patterns! See this for syntax: https://ec.haxx.se/cmdline-globbing.html

Curl can also handle cookies using the -c and -b flags. I wrote a simple dungeon crawler whose user-side "client" is a single line of bash, and the game server is stateless - the entire game state gets stored in the cookie:

  c=x; while [ $c ]; do clear; curl -c k -b k hex.m-chrzan.xyz/­$c; read c; done

(just make sure you don't have an important file called "k" in the directory you run it in, or it will get overwritten)

One great use of curl:

Most web browsers have a "Copy as cURL" function in the web developer menus.

in Firefox it is: tools -> Web Developer -> Network

Under this tab, you will see all the requests used to construct a web page.

You can right click on an item and do "Copy as cURL"

If a web page has some API you want to explore or use in a program, you can recreate it this way.

I often need to get both request and response headers. For this I use:

    curl -vsI https://example.com | grep "^[<>] "

This reminds me of an idea I've been meaning to get to for the longest time, this sequence of exercises would make a great mini-ctf, it would require the addition of a story to motivate it, and an endpoint to provide responses that would contain information to link to the next step. I was planning to do an interactive introduction to the git commands in this style but this topic would work well too. I think it would work well as a medium for self serve training.

I’m learning about Bug Bounty. Burp has a feature properly named “copy as curl command”. It converts the captured request to a curl command, so you can just paste it in the terminal or a file and start playing with it.

After making changes to some argument values in the body, I was getting a weird error. It turns out that I was suppose to also update the content length header. The work around was to simply eliminate that header. But I wasn’t able to find a good explanation as to why.

I would suggest one more exercise; make a GET request to an address where the server replies with a redirect.

I do wish Curl supported OAuth 2.0 out of the box - I would then drop all REST clients in a jiffy.

OK now how does one bypass cloudflare's captcha when attempting to get a blog page or sth? I m trying to create a link-previewer and this constantly shows up

Is there anything to format your responses when using curl ? the reason why I dont use curl often and rely on some browser extensions or an app. for simple usecases

I love using Curl when I get an evil spam email and I really want to see where it’s coming from and what it really does when you click those links.

I find curl to be a great tool, but a poor CLI. Problem is, I'm not sure how to make a great HTTP CLI. Many have tried, but everything I've always seen leaves me wanting.

There are several complications:

  - URI building (things that need URL escaping)
     - path components
     - query parameters
  - headers
     - including common standard ones that should have
       their own command-line options
     - things like ETags that can benefit from storing
       them in xattrs
     - things like conditional requests and range
       requests (think download restart)
  - whether there's a body
     - whether it's an "HTML form" or something else
     - Content-Type
  - whether to chase redirects
  - trust anchors for TLS (HTTPS)
  - what to do with response headers
  - how to integrate filtering of response body vs
    headers (can't use stderr for either)
  - authentication options
  - cookie jars
  - proxies and proxy auth

HTTP is such a complex beast that using it from a shell feels fraught with peril.

For ease of use in a functional way, it'd be nice if, for example, one could set a header value as-is on the command-line, but also by reading from a file (or possibly a pipe, so read only once).

Perhaps something like:

  $ VERB $base_URI [options]

with options for:

  - appending URI local part elements
  - adding query parameters
  - adding headers
  - specifying where to write response headers
  - specifying where to write response body
     - if a file, maybe set [some] response headers
       as xattrs!
  - specifying request body source (default: stdin
    for VERBs that normally have a request body)
  - trust anchors
  - authentication

Examples:

  $ # GET with various options:
  $ GET https://base-URI-here \
      -a local-part-element   \
      -q param value -q ...   \
      -H header value ...
  $ 
  $ # POST with file as request body source
  $ POST https://base-URI-here \
      --file file
  $ # POST with HTML form specified on command-line
  $ POST https://base-URI-here \
      --form field value --form field value ...
  $ 
  $ # And so on for all HTTP methods

Besides the options shown above, which are fairly obvious, it'd be nice to have:

  --rheaders PATH      # where to put headers
  --xheader header     # save given response header
                       # value as an xattr
  -T trust_anchors.pem # or a directory, or whatever
  -C client_cert_and_key.pem
  -U username          # for Basic, DIGEST, SCRAM
  --password-file FILE # for Basic, DIGEST, SCRAM
  -N ...               # For Negotiate.  This could
                       # be key/value pairs that can
                       # be used with
                       # gss_acquire_cred_from()
  -J ...               # Get a JWT token and use it

A convention for using xattrs for storing Content-Type and ETag (and a few other things perhaps) would be very very nice.

I have a massive platonic crush on Julia Evans. This post is yet another reason. So good!

*No, not bicep curls.

I think the name is, technically, cURL. I love cURL and use it all the time. We send our customers cURL one-liner examples of how to test our authentication service which requires POST requests, and it just makes their lives and our lives that much simpler. They can test their stuff very quickly and easily, then ultimately it gets folded into their own services, all debugged and ready to go.