Hacker News Reader

Comment by userbinator | Hacker News Reader

Ironically, not long ago there was an article about how hard string handling is in C, so programs tend to be written to avoid manipulating strings as much as possible. In other languages, where strings are easy to use, it encourages inefficiency like this. The term "stringly typed" also comes to mind.

JSON truly is the new XML, both in terms of advantages and disadvantages. I wish people would stop using it for everything and realise that using a "human-readable" format for data that is 99.999999% not going to be seen by a human is absolute insanity in terms of inefficiency.

View on HN

That's a massive exaggeration. XML is both much more complex (and bug-prone) to parse, and wastes more space.

And the problem here wasn't JSON in the first place anyway, it was

> The “back” stack is unlimited, meaning we’re saving more and more state until we explode

And storing previous state in serialized string in current state, instead of just having it stored natively. You'd have same problem if you tried to serialize <format> into a field in <format>

You'd have same problem if you tried to serialize <format> into a field in <format>

Length-delimited formats don't need (exponentially increasing) escaping.

But why do you bring XML? The parent is against human-readable formats, and XML is one of them.

> And the problem here wasn't JSON in the first place anyway

Yes it was. They stored state in a huge JSON dump which ended up working like a zip bomb by the way they had to escape a string. This alone resulted in an exponential increase in memory that quickly became a problem.

Think about it for a second: would they experience this issue if they stored an in-memory stack per user? They wouldn't.

Heck, it seems that they could live with their JSON choice if they replaced '/' with any other inescapable character and converted it back to '/' when reading.

XML would also have a problem with this: you'd replace < with <, then with &lt;, then with &amp;lt;, … It's not exponential, but still inefficient.

Inefficient is not the same as problematic. They turned what otherwise would be a linear increase in memory usage into an exponential increase, which expectedly resulted in memory exhaustion problems.

There are plenty of problems to go around this approach and overall system design, but the weakest link is undoubtedly how JSON was used.

Can you imagine arguing in a tech meeting that a reason not to use JSON is that it doesn't support unbounded recursive levels of escaping very well?

That is just entirely wrong; not serializing JSON into JSON would turn it back into linear.

I'd argue that JSON is one of the least Human Readable serialization formats.

It lacks comments. Has no way to offer context. Cannot impose limitations, and normalizing needs additional, often poorly supported RFCs (JSON-LD or pointers).

It also is a poor non-human readable format, as it lacks crucial types (datetime, decimal) is rather verbose (just imagine the terabytes bandwidth wasted, globally on all these commas and curly braces) and cannot be parsed streaming.

1. I 100% agree about comments.

2. I’m still not convinced it’s the file format’s job to encode every possible data type. We used to have deserialization layers for that, but for some reason, we don’t do that anymore. Nowadays we expect every file format to be self-describing, which I think is wasteful and limiting. But that’s a matter of opinion, obviously.

3. What do you mean by “cannot be parsed streaming”? Or to be perhaps more precise, what’s missing in existing streaming JSON parsers? I haven’t used any of them yet.

Many JSON parsers require the whole JSON file to be in memory. Parsers that allow parsing in streaming manner are the exception.

I only found jsoncons to work reliable but at the cost of speed compared to something fast like rapidjson. And even jsoncons might have the problem that large strings have to be fully in memory, i.e., if someone has the fun idea to store a 1 GB string inside a JSON, then you still have a potential out-of-memory event at your hand.

There are other attempts to fix this problem with the jsonl format but that format feels so ad-hoc that I'm not convinced yet. It stores one JSON object per line. Of course, this comes with its own problems such as each JSON object potentially still being unbounded and, now that lines are used to separate JSON objects, you cannot format those JSON objects in a human-readable way anymore. Also, the first time I encountered a file like this, it simply had a .json extension and I was wondering why every JSON parser failed to parse that file. Because it isn't valid JSON and should have a separate extension.

Streaming parsers may be the exception, but are likely the correct tool to use when handling large inputs becomes a problem. Similarly, XML parsers evolved the SAX model (as opposed to DOM, where the document had to fit in memory.) It is often sufficient to rip through the file in an event-based fashion to extract what you want, and throw the rest away.

For cases where a single string is truly massive, I would lean towards defining a maximum size to prevent denial of service attacks.

3. Like sibling comment or said: most parsers need the entire file (in memory) before being able to start parsing. XML is just as bad here: both, technically, need the parser to see the closing part before being able to parse the chunk.

The lack of comments isn't really a big deal for serialisation. Nobody is really going to bother adding comments when data is serialised. You could do that with XML if you wanted and I have never seen a single occurrence.

The issue is that JSON ended up being used as a configuration format too, and comments are critical there.

JSON5 is the best answer I have found.

EDN at least provides date literals, UUID literals, and decimals. It also does not require commas to be littered everywhere. Unfortunately EDN is only really well-supported within Clojure. I'm not sure what is meant by "context problem" for JSON, but keywords and symbols in EDN can be namespaced, so that one can distinguish e.g. :human/name and :computer/name. However, it still requires you to write a schema. And due to lack of support in other languages, protocol buffers seem to be a much better choice.

Context being "this branch is a link" or "this string is a creditcard-number" or "this employee here is-a Title-over there, at Company over there".

Then I guess the same is for all these dicts I use in Python.

> as it lacks crucial types (datetime, decimal)

Decimal is there, isn’t it, as part of ‘number’, which lumps together fixed-size integers, IEEE floats and doubles, bigints and bigfloats, and leaves it to the parser to make the right guess as to how to interpret them (often assisted by having the programmer specify a target data type)?

Lumping all those together as ‘number’ makes for a much simpler grammar, but I think that’s not worth it for its negative effects on interoperability. How many json parsers can (easily) read a value of 1234567890123456890E0000000001 and write back the same string to a json file?

Apart from datetime, I’d add duration as a basic type, too, even though that has the problem of being imprecise. Users will want to specify durations such as “1m” or “1y” (is that 365 days? 365.2425 to correct for leap years? Does it depend on interpretation by the user? is it a lunar year of 354/355 days as in the Islamic calendar? Etc)

Agreed. Lack of comments and lack of proper schema enforcement makes JSON a junk pile of a data format.

Definitely not an improvement over XML. Hopefully the next popular standard data format will learn from the past and take the best of previous attempts.

JSON has schemas though. I use them all the time.

JSON itself (rfc 8259) has no schema, nor any mention of it. There are several additional RFCs that allow to add schema or other semantics or limitations, though. But JSON has no schemas.

I think you mean "the JSON standard doesn't define any schema".

Indeed. Nor does it mention any place, extension or addition where a scheme could be defined.

The problem is, that despite JSON's clear and numerous flaws, people still overwhelmingly prefer it to XML. XML is that unpleasant.

But in systems where JSON is used inefficiently, it’s rarely (with some notable exceptions [0]) the a performance limiting factor.

[0] https://news.ycombinator.com/item?id=26296339

That one wasn't even down to json, but rather some c stdlib functions being in linear time because c strings(which then compounded into quadratic) right?

> using a "human-readable" format for data that is 99.999999% not going to be seen by a human is absolute insanity in terms of inefficiency.

Database dumps say hello. :)

If you want human-readable dumps, there's no reason you can't convert the more-efficient format into something human-readable.

I just drag my Windows XML script driver into SYSTEM32 folder (for my HP indigo 7000 series ink press) then I press the rest.

If you want human readable db output, I might introduce you to what I’ve spent most of my career doing: etl. :)

Human readable formats should exist for more or less one purpose, interfacing with a human, eg configuration.

Developing and debugging systems with JSON (vs, say, protocol buffers) is much easier… configuration is rarely interfaced with by humans vs. debug logs or dumps etc, which are always being looked at.

It's not hard to read what isn't "human readable" anyway; it just takes a little bit of practice and learning. For example, I can "sight-read" x86 and Z80 Asm from a hexdump, and bits of ASN.1 and a few other proprietary protocols too.

Did I strike a nerve? It's unfortunate that so many seem to be under the impression that "human readable" is really an euphemism for "like English" and are narrow/closed-minded enough to not consider the possibility that people can learn just about any language with enough exposure and a bit of effort.

4f 6e 63 65 20 79 6f 75 20 73 74 6f 70 20 6c 65 61 72 6e 69 6e 67 2c 20 79 6f 75 20 73 74 61 72 74 20 64 79 69 6e 67 2e

You didn’t strike a nerve, but your comment is misguided.

Of course people can read binary formats. It is just that they read them more slowly, and it requires quite a bit of exposure/practice, so that people usually invest in tooling to simplify the process.

I suggest that when you see a comment like this, you ask yourself “if I change ‘human readable’ to ‘easily human readable’ would I need to offer my correction?”

Your comment comes across the same way as that long ago "Show HN" where somebody said that Dropbox is pretty much just rsync.

With something like protocol buffers, you have an explicit schema, parsing is well-defined, and you rarely (if ever) need to manually inspect the serialized representation.

But json is terrible for configuration, e.g. comments are absolutely required. Toml and Hocon are for configuration.

json isn't human readable, it's a serialization format that happens to use ascii.

Developers are the humans (at least for now) that interface with JSON.

And devs don't care about just config files put API requests and responses and any other kind of structured/semi-structured communication and data.

>using a "human-readable" format for data that is 99.999999% not going to be seen by a human is absolute insanity in terms of inefficiency.

In the encode/decode library we use, the generated toString were accidentally so close to json, that we made them exactly so (mostly had to add quotes). Now we can use json tooling on top of efficient messaging.

Reply on news.ycombinator.com