Hacker News Reader

> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.

What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility.

(Disclosure: I wrote Protobuf v2 and Cap'n Proto.)

Further, like, protobuf now supports a Map<str, str>, although IDK about cap'n or flatbuffers.

That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.

[I am replying to scottlamb and joshuamorton as well as to you in this.]

Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said.

For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions.

proto3 looks like it probably can’t handle this either.

On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million.

Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON.

So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!

I see, the point is that you want to extend someone else's protocol, without their cooperation, and without conflicting with any future protocol changes they make.

That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else.

I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).

> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.

I don't think that's a deal-breaker. protobuf has an extension mechanism. [1] You could have a registry for an org to claim an extension number to a given message (and then use it to hold their own message with as much stuff in it as they like). I'm not as familiar with cap'n proto but assume it has something similar.

That said, I think JSON was a fine choice for this purpose. The efficiency is good enough with standard implementations; and there are much faster-than-standard implementations available if you have a special reason to want one. [2]

[1] https://developers.google.com/protocol-buffers/docs/proto#ex...

[2] https://github.com/lemire/simdjson

Is Messagepack not a useful alternative to JSON in your opinion?

MessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.

> MessagePack and CBOR are equivalent.

MessagePack has an ugly history with text data that's still reflected in some of the libraries interfaces. E.g., the Python library doesn't round-trip strings by default. (They'll turn into bytes objects.) There's a flag to do the right thing, but not everyone is aware of it.

Add to that that CBOR has an RFC, and I'll pick it over MessagePack every time.

Binary formats are mostly nicer from a parsing perspective: it's much easier to unpack. You're right, I think, that simple compression will mostly close any size gap.

> s-expressions have no central specification on precise syntax

There is: https://people.csail.mit.edu/rivest/Sexp.txt

It's imperfect, but it does get the job done. Among its advantages is that it's both pure text and binary: there's a canonical representation which is binary, and an advanced representation which is textual, and they map 1:1 to one another.

Examples:

    (session
     (capabilities
      (urn:ietf:params:jmap:core
       (max-size-upload 50000000)
       (max-concurrent-upload 8)
       (max-size-request 10000000)
       (max-calls-in-request 32)
       (max-objects-in-get 256)
       (max-objects-in-set 128)
       (collation-algorithms ("i;ascii-numeric"
                              "i;ascii-casemap"
                              "i;unicode-casemap")))
      (urn:ietf:params:jmap:mail)
      (urn:ietf:params:jmap:contacts)
      (https://example.com/apis/foobar
       (max-foos-finagled 42)))
     (accounts
      (A13824
       (name "john@example.com")
       (is-personal true)
       (is-read-only false)
       (account-capabilities
        (urn:ietf:params:jmap:mail
         (max-mailboxes-per-email nil)
         (max-mailbox-depth 10))
        (urn:ietf:params:jmap:contacts
         …)))
      (A97813
       (name "jane@example.com")
       (is-personal false)
       (is-read-only true)
       (account-capabilities
        (urn:ietf:params:jmap:mail
         (max-mailboxes-per-email 1)
         (max-mailbox-depth 10)))))
     (primary-accounts
      (urn:ietf:params:jmap:mail A13824)
      (urn:ietf:params:jmap:contacts A13824))
     (username "john@example.com")
     (api-url "https://jmap.example.com/download/{accountId}/{blobId}/{name}?accept={type}")
     (upload-url "https://jmap.example.com/upload/{accountId}/")
     (event-source-url "https://jmap.example.com/eventsource/?types={types}&closeafter={closeafter}&ping={ping}")
     (state "75128aab4b1b"))

Or the binary encoding: {KDc6c2Vzc2lvbigxMjpjYXBhYmlsaXRpZXMoMjU6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29y ZSgxNTptYXgtc2l6ZS11cGxvYWQ4OjUwMDAwMDAwKSgyMTptYXgtY29uY3VycmVudC11cGxv YWQxOjgpKDE2Om1heC1zaXplLXJlcXVlc3Q4OjEwMDAwMDAwKSgyMDptYXgtY2FsbHMtaW4t cmVxdWVzdDI6MzIpKDE4Om1heC1vYmplY3RzLWluLWdldDM6MjU2KSgxODptYXgtb2JqZWN0 cy1pbi1zZXQzOjEyOCkoMjA6Y29sbGF0aW9uLWFsZ29yaXRobXMoMTU6aTthc2NpaS1udW1l cmljMTU6aTthc2NpaS1jYXNlbWFwMTc6aTt1bmljb2RlLWNhc2VtYXApKSkoMjU6dXJuOmll dGY6cGFyYW1zOmptYXA6bWFpbCkoMjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMp KSgzMTpodHRwczovL2V4YW1wbGUuY29tL2FwaXMvZm9vYmFyKDE3Om1heC1mb29zLWZpbmFn bGVkMjo0MikpKDg6YWNjb3VudHMoNjpBMTM4MjQoNDpuYW1lMTY6am9obkBleGFtcGxlLmNv bSkoMTE6aXMtcGVyc29uYWw0OnRydWUpKDEyOmlzLXJlYWQtb25seTU6ZmFsc2UpKDIwOmFj Y291bnQtY2FwYWJpbGl0aWVzKDI1OnVybjppZXRmOnBhcmFtczpqbWFwOm1haWwoMjM6bWF4 LW1haWxib3hlcy1wZXItZW1haWwzOm5pbCkoMTc6bWF4LW1haWxib3gtZGVwdGgyOjEwKSko Mjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMzWiYpKSkoNjpBOTc4MTMoNDpuYW1l MTY6amFuZUBleGFtcGxlLmNvbSkoMTE6aXMtcGVyc29uYWw1OmZhbHNlKSgxMjppcy1yZWFk LW9ubHk0OnRydWUpKDIwOmFjY291bnQtY2FwYWJpbGl0aWVzKDI1dXJuOmlldGY6cGFyYW1z OmptYXA6bWFpbCgyMzptYXgtbWFpbGJveGVzLXBlci1lbWFpbDE6MSkoMTc6bWF4LW1haWxi b3gtZGVwdGgyOjEwKSkpKSkoMTY6cHJpbWFyeS1hY2NvdW50cygyNTp1cm46aWV0ZjpwYXJh bXM6am1hcDptYWlsNjpBMTM4MjQpKDI5OnVybjppZXRmOnBhcmFtczpqbWFwOmNvbnRhY3Rz NjpBMTM4MjQpKSg4OnVzZXJuYW1lMTY6am9obgo=}

This is the equivalent of the JSON:

    {
        "capabilities": {
            "urn:ietf:params:jmap:core": {
                "maxSizeUpload": 50000000,
                "maxConcurrentUpload": 8,
                "maxSizeRequest": 10000000,
                "maxConcurrentRequest": 8,
                "maxCallsInRequest": 32,
                "maxObjectsInGet": 256,
                "maxObjectsInSet": 128,
                "collationAlgorithms": [
                    "i;ascii-numeric",
                    "i;ascii-casemap",
                    "i;unicode-casemap"
                ]
            },
            "urn:ietf:params:jmap:mail": {}
            "urn:ietf:params:jmap:contacts": {},
            "https://example.com/apis/foobar": {
                "maxFoosFinangled": 42
            }
        },
        "accounts": {
            "A13824": {
                "name": "john@example.com",
                "isPersonal": true,
                "isReadOnly": false,
                "accountCapabilities": {
                    "urn:ietf:params:jmap:mail": {
                        "maxMailboxesPerEmail": null,
                        "maxMailboxDepth": 10,
                        ...
                    },
                    "urn:ietf:params:jmap:contacts": {
                        ...
                    }
                }
            },
            "A97813": {
                "name": "jane@example.com",
                "isPersonal": false,
                "isReadOnly": true,
                "accountCapabilities": {
                    "urn:ietf:params:jmap:mail": {
                        "maxMailboxesPerEmail": 1,
                        "maxMailboxDepth": 10,
                        ...
                    }
                }
            }
        },
        "primaryAccounts": {
            "urn:ietf:params:jmap:mail": "A13824",
            "urn:ietf:params:jmap:contacts": "A13824"
        },
        "username": "john@example.com",
        "apiUrl": "https://jmap.example.com/api/",
        "downloadUrl": "https://jmap.example.com
           /download/{accountId}/{blobId}/{name}?accept={type}",
        "uploadUrl": "https://jmap.example.com/upload/{accountId}/",
        "eventSourceUrl": "https://jmap.example.com
           /eventsource/?types={types}&closeafter={closeafter}&ping={ping}",
        "state": "75128aab4b1b"
    }

I think it's pretty obvious which one is cleaner …

> I think it's pretty obvious which one is cleaner …

Yes, clearly it's the JSON. How do you tell at a glance in the s-expr formatted data what's an object and what's an array? What's a keyword and what's a literal? I can barely even tell the heirarchy where it's very visually obvious with the JSON.

I have no strong love for s-expressions, but objects and arrays are programming language constructs, not data format constructs. There's also no way in JSON to distinguish between an int and a float and a decimal type, but that's fine because it's a data format, and not a programming language literal.

The JSON and s-expr have a different indent level, which is why the hierarchy is less obvious to you.

> objects and arrays are programming language constructs, not data format constructs.

Debatable. To me there is a clear difference between a list of things of the same type, and a mapping of names to values of potentially varying types. I don't think the specific programming language really matters, they are two totally different concepts with much different applications.

> There's also no way in JSON to distinguish between an int and a float and a decimal type

True, but they are just small literals which are easy to differentiate at a glance, not big multi-line structures like objects or lists. Plus s-expressions have the same issue.

> what's an object and what's an array?

There are no objects — it's all lists (or arrays, if you prefer). This is a good thing, because ordering is always the same.

> What's a keyword and what's a literal?

Everything's a byte string: any interpretation above that is a matter for the protocol. This is a good thing, because it means you neatly sidestep issues like JSON's numbers actually being floats rather than integers, as well as issues like numeric precision.

'But I want types!' Of course, they are good. But the type of a number isn't just integer or float: it's integer-which-is-a-known-version or float-which-is-between-zero-and-one-exclusive. You need to check those types at ingestion anyway; it's not a big leap from integer-which-is-a-known-version to string-which-parses-as-an-integer-which-is-a-known-version.

> I can barely even tell the heirarchy where it's very visually obvious with the JSON.

They are both indicated with indentation. Probably I should have used two spaces for the JSON indentation, but I was lazy.

Having seen all three examples, I am now convinced that JSON was the correct choice. Thank you for posting all three examples, it's much appreciated. I hadn't seen s-expressions before and I was considering them a plausible alternative, but now that I've seen it, I don't believe that prefix-operator functions in Lisp-compatible syntax are the appropriate way to construct and transmit data from APIs.

> I don't believe that prefix-operator functions in Lisp-compatible syntax are the appropriate way to construct and transmit data from APIs.

Why not? I'm honestly curious, because I cannot comprehend preferring that mess of JSON to the s-expressions. The JSON just looks like a dog's breakfast, while the s-expressions look clean & easy to write, read, parse and use.

Also, I don't think they are functions: they're just lists with a convention. Those lists can be used to expression function calls, as in Lisp, but in a data format I think that the lists more often express structure.

One thing I love about s-expressions is that one indicates the type as the first item in the list, e.g.:

    (foo …)

whereas with JSON one has to invent pseudo-properties:

    {
      "@type": "foo"
    }

(yikes, that verbosity is ugly)

I think what's wrong with JSON is you have to quote the field-names. That makes it unreadable and easy to create typos.

Why are the quotes needed? It would be a simply change in the spec if names without spaces in them could be expressed without quotes.

It's even more puzzling since those quotes are not required in JavaScript...

JSON was created as the lowest common denominator between multiple ancient javascript engines. At one time, at least one browser couldn't handle keywords being used as unquoted object keys, so JSON just has all keys quoted for simplicity. It's the same deal with trailing commas.

You mean JSON, right? (I'm not joking, the JSON looks much more readable and cleaner to me than the S-Expressions)

The S-expressions look much cleaner to me: parentheses and characters, with a few quotes where needed. Meanwhile, the JSON is a mess of curly brackets, square brackets, commas, quotes &c. All the field names are strings, not symbols or tokens. To my eyes, JSON is an embarrassingly ugly format, while S-expressions are clean & readable.

Funny, to me the commas, culry brackets, and square brackets actually make it more readable. They provide context like "this is a list of stuff" or "this is a key:value mapping" which make it much faster to mentally parse it. The S-Expressions on the either hand are a sea of parentheses with no way to quickly tell what's a list or mapping or whatever without closely looking at each item.

Having to quote the symbols is terrible though.

> The S-Expressions on the either hand are a sea of parentheses with no way to quickly tell what's a list or mapping or whatever without closely looking at each item.

I guess that's part of the point from my point of view: it's not about hash tables or objects or even lists, but about what they represent. In my view focusing on the implementation (a hash table mapping strings to hash tables mapping strings to numbers or strings) is looking at the trees and missing the forest.