Hacker News Reader

Comment by nly | Hacker News Reader

A stricter subset of JSON doesn't interest me without more specification surrounding how numerics should be decoded.

How do I decode 18446744073709551615? What is the behaviour of a decoder that can't handle numeric types of that size? Should it cause a parse error? Should it truncate to 1.84467e+19? Should it overflow to 4294967295?

Presumably if you want a canonical textual representation, the parse error is the only acceptable solution.

View on HN

For context, let's start with what JSON is. All the JSON spec really lets you do is say "this sequence of code points is JSON", or "this sequence isn't". It includes a few interoperability suggestions for how you might avoid JSON that will be hard to interpret, but it's extremely stark on parsing and generation.

For a pretty epic summary of this, see this Reddit comment: https://www.reddit.com/r/programming/comments/59htn7/parsing...

You have a comment downthread about how inadequate this is, which I totally agree with:

> The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.

Unfortunately since the goal of Son is to be a subset of JSON, there's not much we can do about encoding/decoding within Son's spec. Instead I hope to build a sane foundation on which other tools can be built. Limiting the scope of Son seems like a good way to go about this. In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers).

EDIT: I'm mixing together two different things in my last paragraph. I don't want Son to know about Floats, Doubles, etc since JSON doesn't. That doesn't mean that we couldn't specify a range on number size though. The reason that is left out is to allow maximum flexibility for other tools to build on Son -- I'm trying to drop extraneous parts of JSON, but as few useful parts as possible. This needs more elaboration within the project's docs.

"In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers)."

If the allowed range of numbers isn't specified, then when you emit data in this format, you can't be sure it will be read correctly on the other end...

This is definitely a problem, but the place to solve it isn't Son.

Son is intended to be a very simple project that grabs all the clear wins. Do we really need both `0` and `-0`, stuff like that.

There are too many different things JSON Numbers can represent for us to have a clear strategy for all them in Son. Int64s, doubles, floats, arbitrary precision numbers (eg https://hackage.haskell.org/package/scientific-0.3.5.2), etc. That makes this a good for for the schema layer (eg JSON Schema or whatever you're using), not the base specification layer.

> Do we really need both `0` and `-0`, stuff like that.

Well, yeah since signed zeros are a thing.

Chrome:

> JSON.parse('-0.0') -0

> JSON.parse('0.0') 0

python:

In [3]: json.loads('-0.0')

Out[3]: -0.0

In [4]: json.loads('0.0')

Out[4]: 0.0

> This is definitely a problem, but the place to solve it isn't Son.

But then what problem does Son actually solve? How is a canonicalisation format that's not actually canonical useful?

You take a hit (e.g. by forgoing fast existing libraries, control over pretty printing, and by being forced to sort on serialization which screws up streaming), but gain essentially nothing, not even the advertised benefit of your stuff not randomly changing as it moves through the stack. The likely main benefit of canonicalization is for security and crypto, but unless you use son without numbers (or create your own subset of son) it's kinda useless for that. Of course you can just encode all your numbers in strings or something, but at the point where you're doing your own number parsing logic (which is the hardest bit), why not just use some well-designed actually canonical format like csexp?

> What is the behaviour of a decoder that can't handle numeric types of that size?

Is this actually relevant? It sounds like an implementation detail to me and it seems to boil down to using arbitrary size bigint. If you can't handle a bigint because it's too big for your machine, it sounds like a problem with the system, not parser.

We can play this game all day. What if I pass an integer that takes 32GiB of memory to store? Also the system's fault if the resources aren't available?

I'd say so. This is kind of a security question - the data is valid from the protocol's perspective, it's the contents that are malicious. It's up to the developer to decide how to handle that - if it's super important to be able to process such big numbers, modify the parser to handle that (for example by storing them on the HDD). Same goes for another serialization format: bencoding. It supports arbitrarily nested data structures and most implementations are recursive, which can easily result in stack overflow if you mess up the implementation. But I wouldn't blame the standard - that's the price you pay for flexibility. At the end of the day, you need to sanitize your input anyway and keep your edge cases in mind. Same goes with XML bombs...

I would think the JS part of JSON would preclude any massive integers. It’s still an IEEE floating point at the end of the day, so your max is (2^53)-1

If you managed to load a JSON string into memory that contains a number that takes 32GiB of memory to store after parsing, I don't think you're worrying much about resources.

The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.

Decode it as a number with exactly that value as the numeric type the application requests. If the number is out of range, indicate that to the application (which should already have logic to handle numbers out of the range it cares about).

If the application doesn't request that you decide it, don't. Leave it in some internal format which does not lose precision. Limit yourself only to memory limits the application requests.

It's no different than decoding a string...

That's great, but most JSON libraries don't give you that level of access... they just decode in to a bunch of nested maps/dicts consisting of your languages native data types. So one day you swap out your JSON library and the behaviour of your application silently changes.

Shouldn't they return a list of warnings/errors on the parse? Just blindly accepting a machine parse of some random input data is how you crash programs with bad input.

Use a better JSON library :)

(There are competent JSON implementations, though they're few and far between. PostgreSQL's is the only good example I can point to off the top of my head.)

Here’s what Rust’s `json` crate does for the `Number` type: https://docs.rs/json/0.11.12/json/number/struct.Number.html

It stores it as positive, mantissa and exponent, allowing you to pull it apart in that way if you need to. Meanwhile, normal usage is via JsonValue (https://docs.rs/json/0.11.12/json/enum.JsonValue.html) which allows you to attempt to convert it into an f64, i32, u8, &c. as you desire, returning `None` if it doesn’t fit inside that type (e.g. -7 won’t go into an i32, 576 won’t go into a u8).

By what’s being talked about here, that looks to be a competent JSON implementation.

Doesn't most of the Rust ecosystem use serde though?

It looks like serde-json does the same: https://docs.serde.rs/serde_json/struct.Number.html

Well, not the same; it eagerly interprets it as PosInt(u64), NegInt(i64) or Float(f64) rather than storing the mantissa and exponent. This is enough for JavaScript compatibility (its Number type being essentially a 64-bit float), but it’s not quite as flexible as the json crate’s approach.

Wonderful. Good to know that about Rust, I haven't used it but everything I read about it looks good. Yes that is exactly the approach I meant (and a similar mechanism to that used by my own C++ JSON library I use for side projects).

This is what unit testing is for. Test the methods that validate and return your expected object structures. This catches other things like a structure changing (as simple as a property being modified), ensuring you can handle malformed input (even if it's valid JSON) and any other data validation rules you have.

Reply on news.ycombinator.com