Son: A minimal subset of JSON for machine-to-machine communication
github.com
github.com
You do this to ensure that documents that contain the same "information" will look identical on the wire, not just after-the-fact once you've processed them into some execution-specific data structure.
For example, X.509 certificates use ASN.1 DER, a restricted subset of ASN.1 BER where each value takes a deterministic form, so that two certificates that contain the same information will look identical when serialized into bytes. This is a stricter application of ASN.1, while, say, when you talk to a directory service over LDAP, you can speak BER, the looser encoding, because the exact bytes by which you're making yourself understood don't really matter.
Son is a canonicalized form of JSON, and the author maintains a page of JSON subsets and supersets [1] that lists two other canonicalized encodings, both of them containing silly flaws in their design. Son is a better effort.
"Object members must be sorted by ascending lexicographic order of their keys."
Looking at the reference implementation, it contains
encode = encodeUtf8 . generate
Relevant line, I think is
genObject hm = "{" <> foldl' go mempty (sortOn fst (HM.toList hm)) <> "}"
⇒ implementations cannot sort the UTF-8 byte st sequences lexicographically. I think that’s a bad choice (if it was a choice and not an oversight)
I’ve never written a single line of Haskell, so corrections welcome.
Agree that specifying the keys to be ordered >by unicode code points< instead of >lexicographically< would be less ambiguous though.
Even if the spec requires it, I fear implementations will also canonicalize strings differently, breaking sort order.
> Object members must be sorted by ascending lexicographic order of their keys.
It also specifies that object keys must be unique, which is another canonicalization issue with JSON.
How do I decode 18446744073709551615? What is the behaviour of a decoder that can't handle numeric types of that size? Should it cause a parse error? Should it truncate to 1.84467e+19? Should it overflow to 4294967295?
Presumably if you want a canonical textual representation, the parse error is the only acceptable solution.
Is this actually relevant? It sounds like an implementation detail to me and it seems to boil down to using arbitrary size bigint. If you can't handle a bigint because it's too big for your machine, it sounds like a problem with the system, not parser.
If the application doesn't request that you decide it, don't. Leave it in some internal format which does not lose precision. Limit yourself only to memory limits the application requests.
It's no different than decoding a string...
(There are competent JSON implementations, though they're few and far between. PostgreSQL's is the only good example I can point to off the top of my head.)
It stores it as positive, mantissa and exponent, allowing you to pull it apart in that way if you need to. Meanwhile, normal usage is via JsonValue (https://docs.rs/json/0.11.12/json/enum.JsonValue.html) which allows you to attempt to convert it into an f64, i32, u8, &c. as you desire, returning `None` if it doesn’t fit inside that type (e.g. -7 won’t go into an i32, 576 won’t go into a u8).
By what’s being talked about here, that looks to be a competent JSON implementation.
For a pretty epic summary of this, see this Reddit comment: https://www.reddit.com/r/programming/comments/59htn7/parsing...
You have a comment downthread about how inadequate this is, which I totally agree with:
> The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.
Unfortunately since the goal of Son is to be a subset of JSON, there's not much we can do about encoding/decoding within Son's spec. Instead I hope to build a sane foundation on which other tools can be built. Limiting the scope of Son seems like a good way to go about this. In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers).
EDIT: I'm mixing together two different things in my last paragraph. I don't want Son to know about Floats, Doubles, etc since JSON doesn't. That doesn't mean that we couldn't specify a range on number size though. The reason that is left out is to allow maximum flexibility for other tools to build on Son -- I'm trying to drop extraneous parts of JSON, but as few useful parts as possible. This needs more elaboration within the project's docs.
If the allowed range of numbers isn't specified, then when you emit data in this format, you can't be sure it will be read correctly on the other end...
Son is intended to be a very simple project that grabs all the clear wins. Do we really need both `0` and `-0`, stuff like that.
There are too many different things JSON Numbers can represent for us to have a clear strategy for all them in Son. Int64s, doubles, floats, arbitrary precision numbers (eg https://hackage.haskell.org/package/scientific-0.3.5.2), etc. That makes this a good for for the schema layer (eg JSON Schema or whatever you're using), not the base specification layer.
Well, yeah since signed zeros are a thing.
Chrome:
> JSON.parse('-0.0') -0
> JSON.parse('0.0') 0
python:
In [3]: json.loads('-0.0')
Out[3]: -0.0
In [4]: json.loads('0.0')
Out[4]: 0.0
But then what problem does Son actually solve? How is a canonicalisation format that's not actually canonical useful?
You take a hit (e.g. by forgoing fast existing libraries, control over pretty printing, and by being forced to sort on serialization which screws up streaming), but gain essentially nothing, not even the advertised benefit of your stuff not randomly changing as it moves through the stack. The likely main benefit of canonicalization is for security and crypto, but unless you use son without numbers (or create your own subset of son) it's kinda useless for that. Of course you can just encode all your numbers in strings or something, but at the point where you're doing your own number parsing logic (which is the hardest bit), why not just use some well-designed actually canonical format like csexp?
For those of you interested, I suggest using Google's protobufs or even full blown gRPC for such endeavors.
See discussion about gRPC here: https://news.ycombinator.com/item?id=12344995
I say that a bit tongue in cheek, but really: BSON has a monstrous amount of basic data types, some of which make very little sense and are very difficult to round-trip through other formats; and MessagePack had crippling basic sanity issues with what-are-bytes-vs-strings in its early history and that's something you simply can't recover from without declaring game over and making a new spec.
CBOR has neither of these issues. It's almost completely isomorphic to json, plus supports byte arrays. It's very nice.
I haven't played with flatbuffers, but it probably supports this as well, given its design. Are there other binary protocols that have first-class (i.e. present with a good API in the client libraries for all major languages) support for lazy/partial deserialization?
EDIT: I am happy to see that it handles date-time. Not really my favorite part of the non-spec of JSON on that front...
They use CBOR as the underlying serialization, and have a deep interest in canonicalization and applications of hashing.
If the issue is debugging, writing a dump tool is a one time effort.
Example: when you add a new element to the API response, the recipient needs to update their schema.
Is that correct? If yes, how do you deal with that in a microservice architecture where you have little control over consumers of your API?
You are free to make forward compatible API changes without a deployment dance.
E.g., you can add a field in the service and the caller won't see it until it is prepared to handle it properly.
... it should never be fragile where fragility might cost more than it gains you. The open/closed principle is the right thing to follow in most cases, but not all.
For example, choking on unrecognized additional fields by default in development is often a big time/bug-saver, since it gives you an early "hint" that things might not be speaking the same dialect to each other.
Similarly, if you have a message convention that is heavy on "modifiers" which control irreversible things, sometimes it's better to be fragile rather than count on orchestration systems to work perfectly 100% of the time. Suppose you have a system for buying bicycles with a required field of "bike model", and modifiers of "color", "wheel size", etc. Now let's say your servers all got patched to expect a new modifier of "frame size" . . . except for one server which didn't restart due to a deployment bug, and is still serving traffic. If a client sends that unexpected additional modifier to the "old" server, and it gets ignored, this could result in sending a production order for an expensive unit with the wrong specifications. If not caught, that cost could be compounded by sending the wrong unit to the customer and pissing them off. Now, this is clearly not a problem with the message format, but rather with the orchestration/release system. But bugs in orchestration layers are incredibly common, and baking a bit of fragility into the messaging layer at the right places can help mitigate those bugs' impact.
That's not the case with protobufs. The sender can add new fields, and they will simply be invisible to receivers using the previous version of the message.
You'd need to update the receiver only if you want it to access the new fields, which is what you'd need to regardless of serialization format.
JSON-schema implementations are also rather widely available.
In performance-sensitive machine-to-machine applications, I agree, better to use someting compiled, but for general-purpose interoperation and archive data storage (where you cannot know when and with what tool you'll want to open it), I've yet to see a compelling alternative.
ProtoBuf depend on the schema definitions and encoder/decoder generation. Without the schema, you cannot read the format (theoretically), if you want your data to be read easily, JSON or CBOR or MsgPack are your best options.
There are few things as slow as non-vectorized MATLAB code. MATLAB's regex engine is pretty decent, actually, and supports quite a few fancy tricks. For example, I use the regex engine to find the start/end of all numbers and strings ahead of time, and for replacing escape sequences (even unicode escapes!) with their literal characters. The former is particularly important because otherwise, you'd have to manually iterate over characters until you find the end of strings or numbers, which is unacceptably slow in MATLAB.
Honestly, that statement applies to pretty much any part of MATLAB unless the behaviour is nailed down by existing mathematical or engineering specifications - and I'm talking as someone that likes MATLAB overall. Mathworks tries a bit to polish and paint it as a coherent whole, but it's patchwork of things built upon and stuck to the sides of the existing structure, optimizing towards a particular kind of "pragmatism" than towards any kind of elegance.
There's also the ability to easily and explicitly encode data that doesn't fit to JSON's limited types. A non-exhaustive example includes: sets, hash tables that don't use string keys, tuples, imaginary numbers, fractional numbers, and precision decimal numbers.
There's also the fact that XML explicitly supports streaming interpolation of documents, something which can speed up interactivity quite dramatically.
The json schema is standardized and available in most languages and platforms.
It's like saying ruby isn't the only ruby language because it isn't a formal standard. So what?
I'd rather use Protobufs or Flatbuffers for specification and then rely on their built-in JSON serialisations.
Remember that the more modern binary protocols like protobuf/etc. don't serve the same niche as JSON schema at all: JSON schema validates external documents; i.e. you take a schema and make sure a document complies with it. Protobuf/flatbuffers/etc. validate internal documents, as they are encoded or decoded. If that's all you need, then the schema definitions for those products will be much friendlier than JSON-schema, but that's comparing non-equivalent things.
Another, minor quibble is that a lot of the schema-validation code in e.g. protobuf or Thrift is often generated by the framework, and the generated code can be quite hairy or confusing. There exist code generators for JSON schema as well, but writing a JSON schema by hand isn't comparable to, say, generating code from the Thrift IDL.
Before looking for something 'better' than JSON, build a basic prototype of your app and see if your application actually needs to use something else. The data in most applications is so small and will parse so easily that actually measuring the CPU utilisation will be quite hard. 99.99% of applications aren't moving data around at the sort of speed where the format really matters.
Protocol buffers are very clever and really cool but using them where you don't need them is a huge waste of effort.
As for the lack of a schema that's not true. There is a schema. What's in the file is defined somewhere. For applications that consume JSON that's in your code rather than the file format.
Plus, the byte count over the wire is much smaller - something that matters even in today's world of gigabit ethernet.
Are you speaking from personal experience? FWIW it doesn't match mine.
We benchmarked one of our apps, written in Go, which uses JSON extensively both for the API and for internal storage, and we found that around 90% of the CPU time is spent serializing and deserializing JSON.
The JSON code itself is a tiny portion of any app, so it's one of those rare optimization cases where a small change can yield significant speed-ups for the entire app.
On the other side, you can produce JSON string directly, saving the need to produce some intermediate representation in memory.
You should be able to saturate 10gbit line on a multicore server easilly with this approach.
That implies that JSON is somehow the gold standard of serialization formats. I don't see it that way. The fact that JSON is ubiquitous in some areas doesn't mean that it's the gold standard for all other uses.
In my experience, protobufs are easy to work with, concise, safe, and efficient. There's never any ambiguity to them, and both creating and parsing messages is fast and easy. That doesn't mean that there aren't cases in which JSONs aren't a better choice, for instance when working with browsers, or needing a text based format for debugging or inspection purposes.
> Protocol buffers are very clever and really cool but using them where you don't need them is a huge waste of effort.
In my experience protobufs require very little effort, with most of the effort being put into defining the schema. It's pretty frictionless afterwards with one notable exception - when you want to manually inspect a message. That's where JSON wins hands down.
I'm not suggesting JSON is the best format, because that depends on the project you're working on. I'm saying it's good enough for most tasks, and has ubituous support, and it's easy to work with. There are better formats for particular use cases but if your application has requirements that can work with JSON it probably isn't worth finding something else.
Contrast with JSON where every decent language ecosystem has a parser readily available. That's what makes it the "gold standard" for me.
No affiliation, just satisfied user.
Please promote this as hard as you can. Then, us old guys getting aged-out of industry will be able to get jobs again.
Make “ASN.2” or something, update the fundamental data types to modern standards (think of what changed from NeXTSTEP to macOS X; they changed from TIFF to PNG, from PostScript to PDF, etc. but kept the overall architecture the same). Consolidate and simplify the many options (but do not eliminate options entirely). And call it… JASN or something, which sounds like both JSON and ASN, and make up some acronym where the “J” stand for “Joint” (like in JPEG, which stands for Joint Photographic Experts Group).
And, maybe most importantly, make freely licensed tooling available for both converting to and from ASN.1 and JSON, but also jgrep, jless, jcat, etc. (similar to zgrep, zless and zcat, etc.) Make these packages available for all common operating systems, readily installable using that plaform’s usual mechanism.
Readable plain-text protocols remove a constant overhead from all debugging/introspection tasks. If you use, say protobufs, you'll still have to convert back into strings to see what was in individual messages. This makes reading, for example, logfiles or wireshark captures with "grep" more difficult. Sure, there are tools that you can throw before "| grep" that convert to text, but that's one more piece of cognitive/debugging overhead you have to remember, and one more thing that new developers have to know to do if they're used to the standard unix/text-oriented way.
That benefit applies even if your messages (or logs of messages) are stored/viewed in a database that speaks your binary protocol and provides tools to search/see the data as text. Whether or not you have something like that, it will always (always) be routinely necessary, or at least incredibly helpful to productivity, to "print out or search the textual content of messages sent/received by my code" when developing and testing apps that communicate over they network. A text protocol has a huge advantage in that department--or rather, it has a very small advantage that is present in a very large number of development/testing tasks.
Also, readable plain text is key. XML dialects, or e.g. Thrift's JSON format are plain text, but are not terribly readable. This is subjective, to be sure, but makes a big difference in practice.
There are great reasons to use binary protocols, but any comparison that leaves out the benefits of text content that is readable "for free" without using per-protocol tools is lacking.
As others have pointed out, text-based serialization has significant overhead. To save a few hours of debugging, you're willing to spend gigabytes of bandwidth, an uncountable number of CPU cycles and memory, and all the associated electricity and hosting costs over the lifetime of your service? Penny wise and pound foolish, to say the least.
Text-based serialization formats are just plain stupid for any type of service that handles over 100 requests per second. A good developer is not hampered in the slightest by the fact that their serialization format isn't text.
Would I switch to a more computationally expensive format to save a few hours debugging each time someone makes substantial changes to my application's message format? Very possibly.
Would I switch to a more computationally expensive format to prevent bugs (the easier it is to view, or remember how to view, what your code is sending/receiving, the less likely it is that lazy people will skip bug-hunting/QA steps that would show it)? Almost certainly.
Should you use a binary format if you're sending tons of data per second? Well, it depends. It's like microservices: if you have the tooling to make dealing with that format from the dev/debug/tracing side a breeze, go for it; using binary provides substantial savings, as you point out. If you don't have the tooling/time to make it as easy as text to introspect, and I mean that in the most absolute way, there might still be persuasive reasons to switch, but know that if you do, you are also buying a lot of "debuggability debt".
(I know there are good reasons for this involving the effort required and maintenance etc. it just bugs me.)
In Windows, PowerShell is a very good example of a low-ish-level (composable commandline apps and snippets) programming environment that isn't shackled by that paradigm.
Ditto. And if you still prefer the schema-free approach of JSON but with faster encoding and decoding support across all common programming languages, take a look at MessagePack.
Unlike JSON, MessagePack is optimized for speed and encodes common data types efficiently. Unlike Protobuf/gRPC, it requires no centralized schema.
MessagePack: https://msgpack.org/
(The other thing SON does is require unique keys, which is a good improvement that fixes an interop bug that can cause security vulnerabilities.)
Could you give an example of a storage format that does do this and the problems that doing so avoids?
Then you also don't have to worry about whitespace, because there is none. The next field can follow directly afterwards, because the type header byte defines how long exactly the current field is. (For variable-length strings, use something like netstrings.)
The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness. You could add a rule like "encode every integer as the smallest possible type", but that may have practical implications elsewhere.
I wouldn't be too worried about it, seeing how half-baked Son appears to be. For example:
> Object members must be sorted by ascending lexicographic order of their keys.
Yeah right, because there is only one lexicographic order in the world.
The specification tells you all strings (keys) must be UTF8. There is only one lexicographic ordering of UTF8 strings (RFC3629, page 2). This should probably be explicitly mentioned in the linked specification though.
Then why is LC_COLLATE still a thing?
Also, you don't get to second-guess a spec. If it does not specify everything that it needs to, it deserves the "half-baked" label.
To be pedantic: Not really. Assuming your example uses twos complement (what else would it use) and you meant "int32" instead of "uint32", negative zero and positive zero would still be the same encoded value. Also since you said "[u]int32", there is only one size the integer can be: 32 bit. So, in your example, the literal value zero has one and only one unique binary representation, which is "0x00000000"
This generally is done for two major reasons: First of all, such an encoding is significantly easier and cheaper to parse than a base10+ascii (human-readable) encoding.
I encourage you to write a parser that reads a fixed, 32-bit binary number (1) and another parser that reads a JSON-formatted number string into an internal variable in a classic language like Java or C++. You will immediately see the big difference in complexity. Make sure your parser can also deal with a message that contains more than just one number, i.e. the parser should be able to tell at which byte index an encoded number begins and ends. Even if you're using a language or library where this is hidden from you (e.g. by using parseInt or std::stod) the same work still happens behind the scenes.
The other reason is that for most numbers and fixed/varint encoding schemes, the "binary" representation will be much more compact. Storing the number "1000000" in base10+ascii (human readable) takes at least eight bytes. Storing the same number in a fixed 32-bit integer encoding takes four bytes. Using a varint encoding scheme might allow you to get down to three bytes.
(1) Ignoring stuff like byte order and representation of negative numbers; this is usually fixed in the protocol/format specification.
Most implementations of json respect that. The error people get are because numbers are converted from double to something else that's not equivalent. Precision is around 15 digits and the usual rounding issues are applicable.
No JSON standard restricts the range of numbers (except for universally prohibiting NaN and infinity, making JSON numbers clearly not IEEE doubles). The standards warns of potential interoperability issues with numbers outside the numbers representable as double, but they do allow all numbers representable in Base 10.
As a specific example, I've had experience in the past where the most common protobuf implementation for Ruby at the time was an order of magnitude slower than the default JSON parser (it has probably improved by now).
So by all means use a binary format for (internal) machine-to-machine communication if the performance gains are significant, but don't just assume that JSON will be too inefficient.
In the 70's, using text formats for protocols was understandable (7-bit serial lines..). Today, it is not.
I mean, I can't think of many situations where I'm debugging something by looking at raw bytes when I couldn't easily just dump the decoded representation instead.
People don't use BCD for ease of debugging do they?
There shouldn't be any remaining string escaping problems though! If you find one definitely open a GitHub issue.
* Support for undef/null
* Support for binary data
* Support for UTF8 strings
* Universally-recognized canonical form for hashing
* Trivial to construct on the fly from SQLite triggers
I'm not promoting it for any particular use case, and I only have a Perl implementation. Posting here as I think some of the audience in this thread may find it useful.
I use HJSON fairly often, in browser and on server, mostly during development.
I'm curious about how often enough trailing zeros occur in real world documents for lack of exponential notation to have a noticeable impact on document size.
+ The diagrams were generated with [GrammKit](https://github.com/dundalek/GrammKit)."
- No exponential notation
That's going to give you some pretty big numbers. It would be better to require exponential notation.
With pre-defined record structure you could simply use arrays instead of objects and save much space.
Please do not make JSON a standard for M2M communication, it's as bad as calling Electron apps "native".