Hacker News Reader

Comment by bastawhiz | Hacker News Reader

> JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more.

If you're parsing to structs, yes. Otherwise, no. Each object key is going to be a short string, which is going to have some amount of overhead. You're probably storing the objects as hash tables, which will necessarily be larger than the two bytes needed to represent them as text (and probably far more than you expect, so they have enough free space for there to be sufficiently few hash collisions).

JSON numbers are also 64-bit floats, which will almost universally take up more bytes per number than their serialized format for most JSON data.

View on HN

> JSON numbers are also 64-bit floats

In common implementations they are, but RFC 8259 and ECMA-404 do not specify the range, precision, or underlying implementation for the storage of numbers in JSON.

A implementation that guarantees interoperability between all implementations of JSON would use an arbitrary-sized number format, but they seldom do.

No idea what ISO/IEC 21778:2017 says because it's not free.

I mean, I suppose you could use a numeric format that takes more than 64 bits, but you certainly couldn't use one that uses fewer. You have no way to know whether a number should be an int or a float. And since JavaScript (the JS in JSON) encodes with 64 bits, you'd be losing precision if you choose to use any less. The point is that an integer written as one byte in the JSON will take up more space (at least seven bytes more) when decoded, all the way up to numbers that take seven characters.

(subsets of )ASN.1, LEB128, and of course UTF-8 are all examples of encoding binary integers with variable length.

Sure, and JSON is as well. 0 takes three fewer bytes to encode than 1234. But if I parse JSON and my numbers come back as a type backed by UTF-8, I'm going to be left confused and angry. Nobody expects their JSON parser to give them back numeric values that their CPU can't natively perform operations on.

After you have spent some time confused and angry you can then read the documentation of whatever library you are using and verify whether it overloads numerical operations for its special number type or it has a function to convert them to whatever numerical type you need.

In any case you wouldn't of course use UTF-8 to represent JSON numbers as it can only encode 21 bit intgers.

I think even structs have this problem because typically you heap allocate all the structs/arrays. You could try to arena allocate contiguous objects in place, but that sounds hard enough that I doubt that anyone bothers. Using a SAX parser is almost certainly the tool you want to use.

Reply on news.ycombinator.com