Hacker News Reader

This isn't some sort of philosophy debate that you're trying to make it out to be. The output of a lot of tools quite simply isn't in a machine-friendly format, and it can be a nightmare to try to write a parser for them yourselves.

You are misinterpreting the Unix philosophy. It's fine to use a bunch of sed, awk, grep, etc. when you're either transforming text or processing already well-structured data. But trying to write a full-fledged parser for something with only human-readable output, especially as a shell script, definitely goes against that philosophy. Congratulations, you've managed to piece together 50 commands in a pipeline and create a monstrosity that's far from the minimalist philosophy.

In fact, I would argue that by using `jc` together with `jq` you can actually create some nice pipelines for parsing the data that will be much more in line with the Unix philosophy.

Nobody ever said this was designed to improve performance, but I have a hard time believing your claims about it being significantly slower which is not backed up by any source. Most likely, eliminating the JSON conversion would be at most an unnecessary micro-optimization. But if your code was truly performance-critical, you wouldn't be piecing it together with shell pipelines that cause a bunch of unnecessary forks, you'd write it in something like C instead.

And the "JSON was designed for the web browser" argument doesn't hold much water either. You're about several decades too late for that, JSON is extremely ubuquitous and used in a lot of non-browser contexts. Sure, some people depending on their needs may use other formats like XML or protobuf, but JSON is still very common.

Yes. The performance point boiled down to this:

> These programs do not expect large amounts of memory and many are written with the intent that they may be used to process text line-by-line

Which is only a problem if you are being very silly, don't choose NDJSON (newline-delimited JSON) and instead shove 10GB of data in a big [] array that the parser has to read in all at once. Almost every single JSON library can do NDJSON already. One of the most heavily used JSON-over-stdio applications is the Language Server Protocol, which uses JSON-RPC 2.0 and is entirely NDJSON. Same for about 15 different log-yeeting tools. Nobody has ever suggested switching LSP to plain text for performance reasons, only lower-overhead binary formats that don't throw out everything gained by having structure at all.

Large memory use by JSON is not something inherent to the encoding that plain text is somehow immune to. All sorts of CLI programs read stdin in all at once, and you don't see plain text getting slammed for exorbitant memory use.

In the context of the original post, `jc` etc, we're talking about essentially a constant sized output that's just much easier to parse, so the complaint is not relevant to those at all.

> But if your code was truly performance-critical, you wouldn't be piecing it together with shell pipelines that cause a bunch of unnecessary forks, you'd write it in something like C instead.

You are underestimating the power of unix tools. A chain of unix tools can match or exceed the performance of C programs written by average programmers. That is a true beauty of unix and partly why it is still relevant today. The author has little idea about performance and doesn't understand how unix works; otherwise he wouldn't make arrogant claims like:

> With jc, we can make the linux world a better place until the OS and GNU tools join us in the 21’st century!

> AFAIK, JSON was designed for Javascript in a web browser

It wasn't. It was _inspired_ by JS's syntax (and that of Python), but wasn't designed for it. Crockford designed it as a lightweight data exchange format that used a familiar syntax. Quoting from the json.org website itself:

> JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language Standard ECMA-262 3rd Edition - December 1999. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language.

JSON isn't terribly difficult to parse, nor does it require "generous amounts of memory". Shy of something like s-expressions, it's about as straightforward as you can get when it comes to structured data.

Netstrings are really useful but they encode strings, not structured data.

JSON in shell might not be faster (or it might be, I've not benchmarked), but it certainly is more efficient to do select and filter and whatnot using something like jq. It's not about network, it's about making the output more predictable when running a script. I've lost count the number of times I've tried to capture a specific part of the output of a command only to be tripped up by an edge case like spaces or something else.

Powershell idea isn't new, it is how REPL worked across all Xerox PARC workstations.

While Windows isn't a whole language OS like those, .NET and COM gets pretty close to it, and that is what PowerShell knows about, instead of raw text.

This is what is missing across most traditional UNIX shells, integrate raw text, UNIX IPC (and newer ones like D-BUS/gRPC), shared libraries, structured data, into a single REPL experience.

In actuality most CLI output is quite small and can easily be represented as a single JSON document. There are a few commands that can produce huge amounts of output and those are good candidates for using JSON Lines as noted in the article.

JC has streaming parsers that lazily output JSON lines for these types of commands. (ls, ping, vmstat, iostat, etc.)

The default behaviour of PowerShell is to stream objects one by one, the same as typical UNIX shells. In principle, it can process unlimited amounts of data on a single pipeline with a small, fixed amount of memory. The exceptions are commands like Sort-Object, which do require everything to be held at once in RAM. In theory, it could do an offline sort like the UNIX "sort" command does, but the issue is that that might break some scripts that rely on .NET objects that aren't serializable. If you're super keen, it would be possible to add this feature and develop a "Sort-ObjectOffline", at the risk that very rarely it might shred some objects...

The problem with JSON is that it does not support streaming by default. It's possible to use non-standard JSON-like formats to work around this, but then you're no longer using JSON!

> The problem with JSON is that it does not support streaming by default.

ndjson is worth knowing about. We use it for things too large to stream.

https://github.com/ndjson/ndjson-spec

"The probelm with JSON is that it does not support streaming by default."

This summarises the problem I have with JSON more succintly. It was not designed for streaming, thus "it does not support streaming by default".

Non-standard, line-oriented JSON formats are usable, although as a user I cannot see how they offer any significant improvement over previous approaches with fewer brackets, braces, colons, commas and quotes (BBCCQ). Consider the BSD utility mtree or the BSD-version of stat. These have options to output text in "shell-friendly" formats,1 minus all the BBCC and excessive Q. Sure, people could add options to utilities to output XML, or line-oriented JSON, but generally they don't. Why is that. Perhaps there is a reason.

You said it best: "It's possible use non-standard JSON-like formats to work around [JSON's limitations], but then you're no longer using JSON!"

Maybe JSON is just about hype or something. An attractant for today's "developers". This would explain why I am just not attracted by it.

1.

https://man.netbsd.org/mtree.8

https://man.netbsd.org/stat.1

A counter example where the utility does output XML is https://man.netbsd.org/envstat.8

There were some upvotes then a series of downvotes as the top comment in this thread changed. Thus, opinion on this issue is mixed.

JSON of course stands for "object" notation. Once we start delimiting the "object" with a newline, or limit its size/length, it arguably starts to sound more like string and the problem of memory requirements is abated.

As the parent comment suggested, when the "object" notation is used for delimited JSON it's really not "JSON" anymore. Is it still describing "objects", or is it describing strings, with the addition of brackets, braces, commas, colons and more quotes. Let the reader decide.

It is reasonable to ask what was the purpose of JSON in the first place, and, if it was designed for sending data over a network, whether previous solutions like netstrings could accomplish the same things as delimited JSON.

for me it is:

readable,

handy since probably every language has libs that work with it fine,

there's a lot of tools that work with jsons e.g generating code classes from json,

it's insanely popular,

really easy to learn

JSONL is the answer for arrays. “[…50mb…]” is too big to be processed in a streaming mode, but particular items usually cannot be split anyway, and “{…}\n50mb more” is what you need.