Hacker News Reader

Comment by phiresky | Hacker News Reader

Would probably be shorter if you used `tostream`

You're right. Good tip:

  jq -r '
    tostream
    | select(length > 1)
    | (
      .[0] | map(
        if type == "number"
        then "[" + tostring + "]"
        else "." + .
        end
      ) | join("")
    ) + " = " + (.[1] | @json)
  '

EDIT: For those who want to save this script, you can put just the jq code in an executable file with the shebang:

  #!/usr/bin/jq -rf

Appending | to each line and a last line . is a jq program that reproduces the original json.

  jq -r '
   ( tostream
     | select(length > 1)
     | (
       .[0] | map(
         if type == "number"
         then "[" + tostring + "]"
         else "." + .
         end
       ) | join("")
     )
     + " = "
     + (.[1] | @json)
     + " |"
   ),
   "."
  '

That's insane. Here I was thinking about how much more challenging it would be to parse and reconstruct the object from jq, and you got the idea to take advantage of the syntax similarity to parse it as jq code itself. Nice. And so, that means the inverse of the jq code I posted would simply be:

  ( jq "$(sed 's/$/ |/;$a.')" <<< '{}' )

As in:

  catj example.json \
  | ( jq "$(sed 's/$/ |/;$a.')" <<< '{}' ) \
  > original.json

And a nice example of the path form being amenable to unix tools.

BTW the input json can be null, so -n works (also using process substitution):

  jq -nf <(sed 's/$/ |/;$a.')

Parsing is awkward in jq, but setpath(PATHS; VALUE) will create necessary structure. PATHS uses the array form, like ["movie", "cast", 5] not .movie.cast[5]. Since 1.5, jq has PCRE regex, so could remove ], and separate by . and [.

Everything you guys have posted in this thread looks like pure ninja magic to me.

looks like ancient Greek for us plebs

This would be more suitable to large json files if used with the `--stream` flag. Here's my take on it:

  jq -c --stream '
    . as $in 
    | select(length == 2) 
    | (
      $in[0] | map(
        if type == "number" 
        then "[" + tostring + "]" 
        else "." + . 
        end
      ) | add
    ) + " = " + ($in[1] | tostring)'

Using `--stream` allows jq to start before parsing the entire json file. In my experience, a 700mb json file can take up 5gb of ram in either jq or python -m json.

You're right about `--stream`, but you didn't need the variable assignment. Also, `-c`, besides the fact that it's not available in jq-1.5 which some people are using, is very pointless in this situation, since we're looking to output text, and `-c` is for outputting objects/arrays in a compact format. The fact that `-r` wasn't used causes jq to output the text encoded as json strings. So, instead of outputting:

  .movie.name = "Interstellar"
  .movie.year = 2014
  .movie.is_released = true
  .movie.else = "Christopher Nolan"
  .movie.cast[0] = "Matthew McConaughey"
  .movie.cast[1] = "Anne Hathaway"
  .movie.cast[2] = "Jessica Chastain"
  .movie.cast[3] = "Bill Irwin"
  .movie.cast[4] = "Ellen \\\\ Burstyn"
  .movie.cast[5] = "Michael Caine"

You're outputting:

  ".movie.name = Interstellar"
  ".movie.year = 2014"
  ".movie.is_released = true"
  ".movie.else = Christopher Nolan"
  ".movie.cast[0] = Matthew McConaughey"
  ".movie.cast[1] = Anne Hathaway"
  ".movie.cast[2] = Jessica Chastain"
  ".movie.cast[3] = Bill Irwin"
  ".movie.cast[4] = Ellen \\\\ Burstyn"
  ".movie.cast[5] = Michael Caine"

Another point is how the strings at the right of the `=` are displayed. They should be quoted. The reason why they're not is because you piped the second element to `tostring` instead of `@json`.

A better version of your suggestion would've been:

  jq -r --stream '   
    select(length > 1)  
    | (
      .[0] | map(
        if type == "number"
        then "[" + tostring + "]"
        else "." + .
        end
      ) | add
    ) + " = " + (.[1] | @json)
  '

The use of `length > 1` instead of `length == 2` is a minor point, but if a future version jq decides to sometimes put 3 elements in these arrays, your filter would ignore those when we're likely to also want those. `length > 1` ensures what we need, that there are at least the elements that we're going to be using, while `length == 2` might filter some of those out, even if it's not right now.

Your use of `add` is neat, though. I wouldn't have thought of that.

Thanks! I've made a gist out of it:

https://gist.github.com/fernandoacorreia/4b67a41bbe227654868...

Reply on news.ycombinator.com