Hacker News Reader

Comment by haberman | Hacker News Reader

XML's bad, JSON is better.

But Protocol Buffers are best. Like JSON, the data model for Protocol Buffers maps nicely onto simple data structures (unlike DOM). But with Protocol Buffers you also get a schema for free, a wicked efficient binary format if you want it, default values, etc.

And you can use JSON as a text format. In other words, you could take your JSON that you have sitting around, whip up a Protocol Buffers .proto file for it, and get nice generated C++ classes for it with full schema validation. Your JSON file:

    {field1="foo", field2=5}

...could be accessed from a C++ object as:

    my_obj.field1();

The only bummer about protocol buffers is that its support for high-level languages (PHP, Perl, Ruby, etc) is not very good. I've been working on a separate implementation of Protocol Buffers in C to address this (by making it easy to write bindings for) but this project has unfortunately been stalled as I've been busy with work and life. :(

View on HN

The protobuf encoding carries a ton of redundant crap around (if not as much as text formats). You can do a lot better by paying more attention to the schema and omitting facts that aren't allowed to change, like tags on self-delimiting required fields and arrays (ASN.1 PER does this well).

The protobuf tradeoffs only make any sense if you think you can somehow do something useful with messages that are somewhat but not entirely corrupted, because you still have to solve the problem of finding intact field boundaries without being given any HDLC-style framing.

> The protobuf encoding carries a ton of redundant crap around (if not as much as text formats).

[citation needed]

One of the features of protocol buffers is that they're backwards and forwards compatible. You can add and remove fields, change "required" to "optional" and back again, and still make sense of what comes to you on the wire. I don't think there's much of anything that can be eliminated from the protobuf binary format.

Eliminating tags for a field (even if you consider it required) wouldn't be backward compatible with a previous version of the protocol that considered it optional.

A message from a previous version of the schema is not "corrupted." Being able to make wire-compatible changes to the protocol is an extremely important feature.

You can't add or remove required fields, because outdated recipients will be dangerously wrong about whether they understand the intent of a message in the revised format. At some point meeting new requirements and keeping interoperability calls for deprecating and replacing the old format, because treating everything as optional (which some Googlers apparently do, avoiding "required" completely) is almost as bad as having 2^n mostly-untested formats.

This is the kind of over-engineering analysis that leads to overly complicated systems like XML schema.

Just because a field says "optional" doesn't mean it's logically optional. You don't have to make your schema formalism complex enough that it can describe every last rule of what it takes for a message to be valid. In fact you definitely don't want to do that, because it's a horrible amount of complexity in the schema for little gain.

Yes, it's true that some Googlers use "optional" instead of "required" everywhere in their .proto files. That doesn't mean that you can omit any field and expect your message to be processed by your peer without error. It just means that you won't get an error at the schema validation level. But the application could still throw an error. More complex rules about what fields must be specified or what values they must have can be described in comments, and enforced with custom validation if necessary.

Also, since protobufs support default values, you can define what value will be returned for scalar fields if no value is explicitly sent. This can often be used to define useful default behavior for the case where a field is omitted.

It comes down to different design choices. Any constraints that are missing from the schema have to be recreated separately in each implementation, and I see them as unlikely to do that consistently enough to interoperate well. So I prefer not to pay at runtime for expressing many variations of the protocol, when I don't expect them to work anyway. Like HTML vs. XHTML—I shudder to think how much work was wasted trying to handle the worst tag soup imaginable, simply for lack of a well-formedness (or DTD validity) requirement.

> Any constraints that are missing from the schema have to be recreated separately in each implementation

Not true at all. The server can implement them -- once -- and any client who makes an invalid request to the server will get an error message. These constraints can be expressed in comments in the interface (.proto) file.

> Like HTML vs. XHTML—I shudder to think how much work was wasted trying to handle the worst tag soup imaginable, simply for lack of a well-formedness (or DTD validity) requirement.

HTML and XHTML is a completely different ball of wax. Insisting on even well-formedness is simply unreasonable in practice, because it is so difficult to ensure, and it is the user who pays the price when the software isn't perfect.

If you're still convinced that the world would have been better if strict XHTML had won, you should read: http://diveintomark.org/archives/2004/01/14/thought_experime...

ASN.1 does not map well to any meaningful and generic in-memory representation (I tried to do that in dfsch's BER/DER module and failed spectacularly), while JSON does, but modifing BER to be efficient json-like representation is not that complex, basic idea is to large extent similar. And as for efficiency and ease of use, if you want efficient dataformat with fixed schema, XDR is certainly better than PER (and free of ASN.1's design-by-comitee nonsense)

I/O is relatively so expensive now that deserializing packed fields is more efficient than transmitting enough padding to make them usable in-place. Some people claim they can do compression and come out ahead on the reduction in sends and recvs and copies.

My take on efficiency of serialisation is more concerned with resulting data size, not actual coding/decoding processing overhead. ASN.1 encodings are mostly concerned about size and not about direct usability of encodings (both PER and BER actually use UTF-8 like encodings for various values, PER even does not align fields to byte boundaries).

And as for compression: dfsch's binary serialization format does some de-facto dictionary compression on some fields and it actually speeds up decoding itself (decoder caches various high-level metadata in decompressor's dictionaries), not only reduces I/O size.

What is it the tool that generates C++ codecs for JSON from .proto specifications?

(I wrote a generic command-line Protobuf-JSON converter).

Hmm, I just looked and apparently this code isn't part of the open-source protobuf release. :( This code exists inside Google, but apparently hasn't been disentangled from the rest of the Google source tree to be released outside.

Reply on news.ycombinator.com