Jolt – JSON to JSON transformation

> I was hoping jolt was a command line tool in the style of jq. I dont believe jq does transformations of json, only queries.

I guess that depends on one's definition of transformation but:

    echo '{"hello": "world"}' | jq '{brave:{"new": .hello}}'

it also doesn't even require JSON input, meaning one could use it strictly for composing valid output JSON:

    # where -n = null input
    jq -n --arg foo bar '{"meet me": ("at the "+$foo)}'

I use it for transforming AWS responses all the time. I put jq up there with vim as one of the "must have" tools. In fact, the keys `%!jq .` are burned into my fingers because I type them a lot

_ed: fixed a mobile character replacement, added in the --arg flag I couldn't remember at the time_

You should rather look at App::RecordStream. Last time I compared them jq was a poor cousin of App::RecordStream.

It does! And you can do wonders with it. It has functions that make it capable of doing almost anything.

Also, my project https://requesthub.xyz/ relies entirely on the transformation capabilities of jq. See real world examples of jq transformations in this context: https://gist.github.com/fiatjaf/1d57953aa285b9bf5b51712268cf...

I did XSLT years ago and had far less of a problem learning my way around stylesheet syntax, and that was that bad old XML, don't you know. I was looking into Jolt just a week or two ago and ended up having a really rough time. I think if there was a more deliberate tutorial, it might be easier to learn. But the Javadocs and the hairball of example transformations were something I found confusing.

And DSSSL before that. But I guess XML is just not hip enough anymore.

Actually, having done some work on XSLT these days again I was joking to a colleague at work that one day someone will surely think what a great idea it would be to create a programming language in JSON to transform JSON to other JSON. And lo and behold, a few days later I stumble over this.

I have to admit, XSLT looks far nicer to me, but that might be familiarity and a proper specification to read to understand things instead of just a bunch of examples.

Turtle is a lot more popular than N3:

https://www.w3.org/TeamSubmission/turtle/

Turtle is a subset of N3. N3 has some syntactic sugar plus also some syntax to describe first-order logic statements that include implication and the use of existential and universal qualifiers.

Tim-Berners Lee started work on this software:

http://infomesh.net/2001/cwm/

which implements reasoning on N3, but it is not particularly performant for a number of reasons -- one is that it does not implement any description logic optimizations, another is that it is written in Python, another is that it doesn't have a particularly sophisticated rules engine.

The closest thing to a standard in this area is ISO Common Logic

https://en.wikipedia.org/wiki/Common_Logic

which is based on KIF but uses RDF for the basic data structures, i.e. you can load RDF data and work on it with Common Logic.

There are plenty of people who use logical reasoning over RDF data but usually they use something specific to their tools, such as the Jena Rules Engine, or some kind of Prolog -- it also is not that hard to stuff RDF data into a SAT solver, theorem prover or similar tools.

It's a conspicuous absense that there is no common standard for production rules.

It is a big mistake to say that RDF/semantic web has "failed".

What has happened is that people have tried a lot of things and some have succeeded and some have failed. For instance, OWL seems to have become an inpediment to progress rather than a useful tool. On the other hand, people are starting to understand what JSON-LD is and they like it.

See https://developers.google.com/schemas/formats/json-ld

Also note that by simply specifying a namespace to put the terms in you can transform JSON into usable RDF and even put multiple documents in and see them as a comprehensive graph.

I paste a previous comment of mine, from an old HN thread: https://news.ycombinator.com/item?id=10947013 "RDF has no adoption. JSON is the winner. Let's try to understand why: Because JSON.parse() is instantly available on the client. On the contrary, RDF.parse() (or more precisely N3.parse()) is available on the client only if you use that obscure library I mentionned above. Noone knows the library, so noone uses the N3.parse(). Ok. End of discussion. Now my own feeling: RDF (and especially its N3 dialect) is the only description language I know of that serializes and deserializes graphs with no specific code, and has a good JS lib for client-side consumption. JSON (or XML, or any other tree descrption language) requires extra effort whenever you use it to serialize/deserialize/traverse a graph structure. That is why i think they suck. And that's why I prefer RDF/N3."

But without a standard library

But but but it's FUNCTIONAL, and turing-complete!!!

// yes I understand the use cases of functional code

Increase the terror further by adding Jsonx to the mix: http://www.ibm.com/support/knowledgecenter/SS9H2Y_7.5.0/com....

If you like XPath and JSON, too, you can use my just updated Xidel (http://www.videlibri.de/xidel.html) to run XPath queries on JSON.

It can do transformations, where you write the output structure, and then pick the values from the input JSON. E.g. to transform from {"a": [{"b": 123}, {"b": 456}]} to {"c": [123,456]}, you could write an XPath expression {"c": a/b }

> But in general, JSON, XML, S-expressions, yaml, HTML, and many data interchange programs are all 95% the same hierarchical data structure.

S-expressions can also represent graphs. I have a vague recollection that YAML can as well.

An S-expression example:

    (nodes #1=(a #2=(b #1# #3=(c #2# #3#)) #3#) #2# #3#)

This cycle keeps re-occurring in multiple places every few years because the ideas are in fact useful, but the surrounding ecosystem falls out of favor.

Old solutions accumulate cruft and start to need more expertise (and sometimes, straight wisdom) to operate well. People complain, blog posts are written, inertia and inaction are challenged; babies are thrown out of the bathwater as clean-room rewrites commence with different stacks, different tradeoffs, and often, different bikeshedding.

The new, shiny tool is promoted on its merits; some of that excitement inspires hype, which takes on a life of its own. Through a combination of informed usage and less-informed codemashing, the tool accumulates users from all walks of life, who turn into willing or unwilling stakeholders in its future. Development pressures pull the product in different ways, some people leave, eventually only the really passionate or really locked-in remain. These people acquire domain knowledge, "sometimes straight wisdom", making the solution -- its merits, design rationale, tradeoffs, and lessons learned -- more opaque to newcomers. Those newcomers find that existing solutions don't appear to do what they need at the level of complexity that they can grok. The cycle repeats.

> a very simplistic serialization format with a minimal number of built-in structures.

It's already here: protobufs.

https://en.wikipedia.org/wiki/Protocol_Buffers

Yup. There is an insane difference between how straightforward it is to transform data structures in Javascript and how cumbersome it is in Java.

> it's a lot easier to find a Java developer than it is someone who knows Groovy, Rust, Scala, etc.

By mentioning Apache Groovy in the same breath as Cobol, Java, Rust, and Scala, you're comparing apples and oranges. Groovy is to the JVM what bash is to Linux. Groovy is a dynamically-typed language, best used for writing scripts used for glue code, testing, build control, and what not, whereas those others are statically compiled, used for creating actual systems. Although static compilation was added later, virtually no-one uses it.