Hacker News Reader

Comment by protonfish | Hacker News Reader

I wish this (and JSON) wouldn't be ambiguous about number type (integers vs. floating point.) There are very different types of data in my opinion.

View on HN

A numeric tower could be useful. https://en.wikipedia.org/wiki/Numerical_tower

C R Q Z N Complex, Real, Rational, Integer, Natural

This comment has generated a lot of interest, and rightly so. The thing to remember is that Ren (and JSON) are data exchange formats, not in-memory representations. If you need more control than their numeric syntax supports, metadata is your friend. Just include that where needed. That doesn't mean every endpoint will honor or treat them the same, of course.

"...are data exchange formats, not in-memory representations."

Real world values are still typed.

"If you need more control ... metadata is your friend."

aka implicit user typedefs. Explicit is better.

Explicit is different, not better.

They are different data types, but integer is also ambiguous, did you mean signed or unsigned, 32 or 64 bit. Its about your use case, and when your parsing something with loose typing into stricter types, make sure you validate accordingly. But being able to specify the exact type kind of defeats the purpose of an interchangeable data format.

Yes, the specific type of integer and floating-point can vary, but how operators work and display formatting should be consistent. For example, Modulo should be integer-specific, and the concept of precision only applies to floating-point numbers.

> integer is also ambiguous, did you mean signed or unsigned, 32 or 64 bit

These are all integers. There only needs to be one integer type for all of: 0 6 -6 8000000000000000000000. Python gets this right.

Now 90%+ of the programming language communities have a bar to implement your specification, because they have to emit a weird type out of the deserializations that will break code in weird ways, and in the case of static languages, you may have created a deserialization format that always emits BigInts (several words of memory and possible a mandatory indirection based on implementation) and will thus be unable to compete on the benchmarks against the serialization formats that specified integer sizes. And your serialization format is that much closer to withering on the vine with no usage.

Or you can go the JSON route, mumble your way through the numbers spec, let every language do its own thing, and tell the handful of people seriously interested in moving integers too large to be precisely specified by a 64-bit float to encode as strings or something and stop worrying everyone else with the complexity....

> in the case of static languages, you may have created a deserialization format that always emits BigInts

I don't follow this one. If your language, static or otherwise, is capable of turning this JSON:

    "100"

into a string, and this:

into a number of whatever type, then it's also capable of turning this hypothetical input:

into one type of integer, and this:

    100000000000000000000000000000000000000000000000000000000

into another type of integer.

Not really; those int types differ in range, but while that matters for a schema format, it doesn't really for an interchange format.

OTOH, exact decimals (of which integers are a subset) differ fundamentally in meaning from limited precision binary (or decimal, though that's more rarely encountered) floating point approximations. Which matters in interchange as well as schema.

Because JSON is a text format, nobody prevents you from encoding floats like this:

{ "int": 1, "float": 1.0 }

Of course it's up to the parser implementation to interpret 1.0 as a float.

The spec is very relaxed about the number type:

https://tools.ietf.org/html/rfc7159#section-6

I disagree. For 99% of applications, a number is a number is a number. Integers and floating points are a leaky abstraction, I expect the language/compiler to handle these.

I'd put it at more like 50% of applications.

Try passing around 64bit integers in json. The majority of json implementations only implement double float numbers and will mangle your 64bit ints. The usual solution is passing int64s as strings, nasty.

Those implementations are as broken as a database that stores ZIP codes as integers.

Hit F12 on chrome and enter the following:

    JSON.parse('[9223372036854775805]')[0] === JSON.parse('[9223372036854775806]')[0]

You should get:

    <- true

Is chrome's implementation broken? Those are both numbers that a signed int64 can store precisely but a double float cannot.

Yes, the JavaScript JSON parser is broken (in design), in that it silently loses data/precision of JSON numbers. JSON numbers are not "int64s" or "double floats", they are arbitrary-precision decimal values. [1]

Postgres is the only popular system with built-in JSON functionality that I know of to correctly round-trip JSON numeric data. Python comes close but fails for any number with a decimal point or in E-notation.

One can argue that the JSON spec is too permissive (and I would disagree, though I'm somewhat a purist). Or a pedant could note that the JSON spec doesn't actually say whether any of the digits in a number are considered to carry information (and does in fact note that many parsers are faulty). But it's unfortunately true that most popular JSON parsers fail to round-trip valid JSON data due to flawed design.

[1] http://json.org/

> Or a pedant could note that the JSON spec doesn't actually say whether any of the digits in a number are considered to carry information

RFC 7159 not only specifies that all of the digits carry information, it specified exactly what information they carry.

However, it also expressly permits implementations to limit the range and precision of numbers accepted, recommending (but not requiring) range and precision at least equivalent to IEEE 754 float64 be supported.

> RFC 7159 not only specifies that all of the digits carry information, it specified exactly what information they carry.

Can you quote? I don't see where it does, except by reference to common knowledge.

But I should clarify. There are several inefficiencies in JSON numeric representation which may or may not be considered significant by an application. The ones I can think of, in order from "obviously not" to "well, maybe":

1. the case of "e" vs. "E"

2. the optional "+" sign after "e" or "E"

3. presence/absence of decimal point and/or E-notation (shouldn't make a difference, but does in many parsers, such as Python's)

4. the value of the exponent itself (e.g. 3.14 vs. 314e-2)

5. "-" sign in front of any number with a value of 0 (IEEE floats and ones-complement integers do have a negative zero)

6. excess trailing 0 digits after the decimal place (may be used to represent significant figures in scientific applications)

7. digits of lesser significance (obviously the most contentious)

JavaScript ignores all but #5. Python ignores all but #3, and due to #3, sometimes #5 and #7. PostgreSQL `json` ignores none; `jsonb` ignores all but #6 and #7. Personally I would draw the line between #4 and #5. But neither the RFC nor the ECMA spec tell us.

One of the multiple JSON specs indicates that implementations are free to introduce limits on precision or range of numbers: https://tools.ietf.org/html/rfc7159#section-3.

I concede that "broken" is too strong a word. "Subpar" would be more accurate. Like a 7-bit MIME gateway in an 8-bit world.

No, JavaScript simply has inadequate numeric support.

Chrome’s implementation is fine (within spec) and makes the natural choice for a general purpose JSON implementation for JS.

Yes, I think everyone is in agreement that Javascript is broken.

That's not a JSON problem, that's a problem with whatever language you're using to consume the JSON. Back to my point, you as a developer shouldn't have to worry about this at all. You should be able to add numbers to JSON and retrieve them without it harassing you. To a human, 123, 0.123, 12345789347590837452987543758973459734 are all the same "type". Why is it that you can save your int64s as strings but not as numbers? It's a stupid leak on the part of the language if you have to use this workaround.

> Why is it that you can save your int64s as strings but not as numbers? It's a stupid leak on the part of the language if you have to use this workaround.

Javascript, for example, has this problem because everything is just a "number" and the number data type is a IEEE-754 double precision float. A double precision float can't represent all int64 values, so transmitting an int64 via JSON to Javascript and using the standard JSON parser is lossy. You could say that Javascript screwed up by making number have a specific precision, and you might be right. Python, for example, gives you seamless arbitrary precision arithmetic.

But the decision to use an arbitrary precision type for all numbers in every language is definitely not appropriate. Arbitrary precision has trade offs, and sometimes it's important to work with machine integers.

The correct solution is to only convert a JSON value to an integer (or string, or array, or dictionary/object) when the application explicitly requests it. 99% of the time, 64-bit integers are not used by a JS app directly, but rather are passed right back to the backend they came from. If they were never converted to JS numbers in the first place, there can be no precision loss.

"Deep embedding" of JSON data into the host language is the cause of this issue. Recognition that JSON is a separate data type and does not always have an obvious encoding in the native language (including JSON arrays, objects, and null, even if they look like native arrays, objects, and null when you squint at them) is key.

The only widespread JSON implementation I know of to get this right is Postgres's: JSON values live as the "json" or "jsonb" type until you explicitly convert them. No data loss is incurred, with the exception that "jsonb" normalizes E-notation, conflates 0 and -0, drops duplicate object keys, and normalizes their order. Even numeric significant 0 digits are preserved.

The ideal solution is to parse all numbers as a decimal or rational.

(one of my favorite things about Perl 6 is that non-integer literals are rationals by default; floating-point is only used if the literal is specified in scientific notation)

Reply on news.ycombinator.com