Bits on the Wire

Much like type systems in general, you always have a schema, and you always have to worry about what happens when someone adds a field. You just get to choose whether it's explicit and you have to type it alllll the way out allllll the time, or whether it's implicit, and is defined as "whatever all the usages are everywhere".

Just as gradual typing is taking over the world (TypeScript, MyPy, etc.), I predict success for something that lets you gradually add schemas to JSON.

Useful context for the youngsters: Tim invented XML. I mean, techically, there were a few other people who helped write the spec, but Tim was using more-or-less XML in shipped applications before they got involved.

In my book, that means his opinion about the importance of bits on the wire is pretty weighty, but you also shouldn't discount the possibility that he's in some sense constitutionally inclined toward standardized data formats.

About other data formats:

I have some experience with CBOR which I regret; it's more compact than JSON, but it's usually not significantly more compact than gzipped JSON, and it's usually slower than JSON, which is saying something, and is surprising given that no scanning for delimiters is needed. Its only real advantage is that you can safely include binary blobs. It's worth noting that originally CBOR was part of MessagePack until its author decided to fork it off and name it after himself, Carsten Bormann.

Cap'n Proto, FlatBuffers, and SBE (which I guess is too obscure for Tim to mention) get their "deserialization in negative nanoseconds" by virtue of not deserializing; instead, the respective libraries access the desired fields of the serialized data in place, which involves an extra pointer indirection or two and possibly some byte swapping, so it's a little slower than accessing fields of native structs. But it means you don't have to parse the entire data structure, just the parts you're accessing, so you can do things like memory-map a read-only data file instead of parsing it. In many cases, even if you're accessing the whole thing, saving the extra memory copy is actually worth more performance than avoiding the byte-swapping and indirections. This is not a new idea — COBOL was based on it, though without these systems' extensibility and consequently without the cost of the extra indirections — but it's an extremely effective idea in many contexts.

All of FlatBuffers, Cap'n Proto, SBE, Protobufs, Thrift, and I think Avro do have support for adding new fields, contrary to what Tim says about binary formats in his note; but they mostly do presuppose that those new fields are being defined by some central authority, because there are things like sequentially assigned field IDs, so certainly you don't have the level of safe extensibility that you have in JSON (or HTTP or RFC-822 mail or HTML forms or SNMP ASN.1 BER data or, of course, XML and HTML tags and attributes).

It has one advantage over YAML though, with JSON you know when you're not at the end of the file :-)