A list of command line tools for manipulating structured text data

It is, in a sense, liberating to not have any library support for common problems: No need to learn a library (hej!) and, by the way, what you need in any given problem is an easy subset of what that library would do anyway.

Awk has made me focus more on the data there is to analyze, rather than the framework to analyze it with.

As the idiomatic use of awk is also very succinct, I hardly can imagine working efficiently on the command line without it.

In the days perl was more commonly used inline ( perl -e 'print "\x90"x200;' and friends) it was often referred to the same way.

As are similarities with some features of Gema, in some cases striking. They are complete coincidence; I had no awareness of Gema whatsoever when I started TXR in 1009. For instance, the use of the @ character. Check this out: Gema has @bind{var;string}. TXR has @(bind var expression). Complete, utter, double-take, rub-your-eyes coincidence! The bind terminology in TXR comes from Lisp and from pattern matching and unification concepts: variables in patterns bind to the parts of the data which they match. So of course if you introduce an explicit construct for extending the bindings, you call it bind).

TXR is a project with a much larger scope than the current state of things like Gema and TXL; it provides a complete programming language that is paired with a text extraction language.

This is from the ground up, pretty much, emanating from the way it is parsed in its entirety before being executed.

You can develop the logic in small pieces and test them in isolation on some sub-segment of the data, then integrate the pieces into a larger script with @(load ...).

Speaking of code refactorings, I also use it for that myself. When using it to interactively rewrite code, I invoke it in a pipeline out of vim. E.g. select some range of code visually with V, then "!txr script.txr -[Enter]" to filter it through. Vim deletes the original text and replaces it with the output. If the output is wrong, I hit u for undo to restore the original text. Then of course I have to fix script.txr and save it, and recall and repeat that filtering step just with ":[up arrow][Enter]".

To avoid refreshing the output file during development, don't have one; let it dump to standard output.

Something useful is that we can also monitor the first N lines of an output file that fit into the screen using the watch utility that is found in many GNU/Linux distros. Watch repeatedly executes some command that is given (once every 2.0 seconds by default) and splashes its output on a clear screen. If we "watch cat file", we can monitor the changing contents of file.

http://www.kylheku.com/cgit/tamarind/tree

This is a CGI script that runs under Apache. It doesn't use any web framework; everything is there in the code: parsing URL parameters and form data, handling sessions and authentication, etc.

In early versions, I scanned the /etc/passwd file and authenticated users by doing raw crypt calls on the passwords. Then I upgraded to SASL authentication: sending tokens to "saslauthd" over Unix sockets. This is in

http://www.kylheku.com/cgit/tamarind/tree/auth.txr

Then I added IMAP. That was an example of incremental development. I tested the IMAP code in isolation and then just planted it into auth.txr.

I was surprised because I was sure that would be done in Lisp; but the pattern language easily implements the IMAP conversation needed to authenticate.

I expected most of Tamarind to be Lisp; I'm surprised how much of it in general ended up TXR via a path of least resistance.

Interesting, I didn't realize Clojure had a reputation for data transformation and extraction

  (let [; input
        js (cheshire.core/parse-string (slurp (clojure.java.io/resource "awful.json")))
        ; utils
        get-match (fn [k v coll] (first (filter #(= (get % k) v) coll)))
        model-for (fn [title toplevel] (get-match "title" title (toplevel "models")))
        key-for (fn [name* model] ((get-match "name" name* (model "fields")) "key"))
        ; models
        fruit-model (model-for "fruits" js)
        farmer-model (model-for "farmers" js)
        ; keys
        fruit-name-key (key-for "Name" fruit-model)
        farmer-name-key (key-for "Full name" farmer-model)
        farmer-fruits-key (key-for "Fruits" farmer-model)
        ; values
        bob-entity (get-match farmer-name-key "Bob, the farmer" (farmer-model "entities"))
        bob-fruit-keys (bob-entity farmer-fruits-key)
        bob-fruit-entities (map #(get-match "_id" % (fruit-model "entities")) bob-fruit-keys)
        bob-fruit-names (map #(% fruit-name-key) bob-fruit-entities)]
    bob-fruit-names)

  => ("Orange")

    (let [models (->> (get (clojure.data.json/read-str json-string) "models")
                      (clojure.walk/keywordize-keys))
          fields (->> models
                      (mapcat :fields)
                      (map (juxt (comp keyword :key) :name))
                      (into {}))
          entities (->> models
                        (mapcat :entities)
                        (map (juxt :_id (fn [entity]
                                          (->> (for [[k v] (dissoc entity :_id)]
                                                 [(get fields k) v])
                                               (into {})))))
                        (into {}))
          ]
      )

    .models
      | (.[]
        | if .title == "farmers"
          then
              (.fields | .[] | if .name == "Fruits" then .key else empty end)
                as $fruits
            | (.fields | .[] | if .name == "Full name" then .key else empty end)
                as $name
            | .entities
            | .[]
            | if .[ $name ] == "Bob, the farmer" then .[ $fruits ] else empty end
          else empty end)
            as $fruits
      | .[]
      | if .["title"] == "fruits"
          then
             ( .fields | .[] | if .name == "Name" then .key else empty end) as $fruit_name
             | .entities | [ .[] | {(._id): .[$fruit_name]} ] | add as $lookup
             | $fruits | .[] | $lookup[.]
          else empty
        end

    data=input.json
    
    fruits () {
        jsonaxe 'models.filter(lambda x: x["title"] == "fruits")[0]' "$data"
    }
    
    farmers () {
        jsonaxe 'models.filter(lambda x: x["title"] == "farmers")[0]' "$data"
    }
    
    fruit_name_key=$(fruits |
        jsonaxe --raw 'fields.filter(lambda x: x["name"] == "Name")[0].key')
    
    farmer_name_key=$(farmers |
        jsonaxe --raw 'fields.filter(lambda x: x["name"] == "Full name")[0].key')
    
    farmer_fruits_key=$(farmers |
        jsonaxe --raw 'fields.filter(lambda x: x["name"] == "Fruits")[0].key')
    
    farmers |
    # get bob
    jsonaxe "entities.filter(lambda x: x['$farmer_name_key'] == 'Bob, the farmer')[0]" |
    # get bob's fruits key
    jsonaxe --raw "get('$farmer_fruits_key')" |
    while read fruit_id
    do
        fruits |
        jsonaxe "entities.filter(lambda x: x['_id'] == '$fruit_id')[0]" |
        jsonaxe "get('$fruit_name_key')"
    done

That's a good quote