Comment by candiodari | Hacker News Reader

Comment by candiodari | Hacker News Reader

Parent Full thread

Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front.

* cap'n proto https://capnproto.org/

* flatbuffers https://google.github.io/flatbuffers/

* hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/

You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in.

Protobufs focus on:

* standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from)

* streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory)

* extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand)

* language agnostic

* composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable)

* an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?)

And in this protobuf will beat the above formats (except cap'n proto).

None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.

All of those significantly post-date protocol buffers, though. It would have been great if they had existed before PB, so we could have used them at Google. But, we had to invent something because nothing that existed at the time was suitable, and the migration cost to use something else now would be astonishing. I'll readily admit that a new company would likely be better off standardizing on one of these. However, standardizing on Protocol Buffers is still way better than having a mish mash of different formats, even if all of those formats are individually better.

Would you recommend someone, not Google, to standardize on protobuffers now?

I'd probably pick Cap'n Proto or Flat Buffers if speed were paramount, in a grass is greener sort of way. I haven't used either of those technologies, though, just read about them. I'm also cool with plain JSON, which is beautiful from an ease-of-getting-started and universality perspective. I also think GraphQL is super compelling, and there's something to be said for records-as-in-SQL.

Mainly, I just think that interchange formats should be boring, simple, and ideally not incredibly slow. Protocol buffers at least meets those bars, even if it doesn't meet the consistency bar that the article desires. The whole "stop being a hipster and just use X" meme comes to mind. Probably realistically X is JSON in this day and age.

Absolutely no to JSON, IMO.

The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code.

Figuring out these as hoc formats is nigh impossible, so you end up just looking at the data over the wire and writing code that parses what you see. Except then you have broken code that doesn’t parse the less-common variants that have extra fields, don’t have extra fields, sometimes have their fields as stringified ints instead of just binary ints, etc.

Virtually every time I’ve seen JSON used as an internal interchange format, reverse engineering that format a few years later.has become a massive, error-prone, tedious, and time-consuming task. Save yourself from this ahead of time and pick formats that require a predefined structure, like protobufs.

This is the classic static vs dynamic typing question. Suffice to say, I don't think there is sufficient science or anecdata on the issue to really give a solid answer.

I am a proponent of both static (Rust) and dynamic (Ruby) typing in programming languages. They both have their place, and if there are bugs you can always fix the code.

The one place where I don’t think dynamic types have their place is in internal interchange formats. These can live forever, and they can change subtly over time. It’s the living forever part that changes the calculus.

This.

Don't forget integer type defined by implementation. Very easy to shoot yourself in the foot in when consumers/producers are not written in the same language, even more if it's a mix of dynamic and static languages.

Nothing beats a well defined and commented schema.

To be more precise, this isn't really a knock on JSON so much as the culture surrounding people who typically use JSON.

Presumably you would be okay if the care and rigor were taken to define a schema and its physical representation just happened to be JSON?

How many of those people exist?

Protobufs also run anywhere, there are libraries of it ported to almost any machine.

When my team was deciding on a binary serialization and RPC communication format, we needed something that ran on a Cortext m3 with no malloc, and every major phone platform (including WP at the time) and desktop OS.

It came down to cap'n proto, and Protobufs.

Protobufs runs everywhere. It won.

Am I missing something, or do cap'n proto and flatbuffers not support mapping types?

Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal

Protobuf did not support maps for a long time. In practice using a list of key/value pairs works fine for most use cases. Sending the message over the wire is obviously always O(n); if you need fast inserts/lookups in code, then it usually makes sense to convert to and from the appropriate in-memory data structure for that purpose.

That said, I think maps are a fine feature for a serialization format to have. But, I haven't gotten around to adding them in Cap'n Proto because I have yet to hit a use case where I wasn't happy with a list of key/value pairs.

For what it's worth, I like them primarily because I don't mind using proto-generated objects as pieces passed around the stack (at least at higher levels of the stack). I lose consistency if I can do that for some objects, but am required to convert objects if they would need a mapping type.

It follows sorta the same reason that Clojure became a fairly widely used lisp – mapping types are extremely common in real world programs. Having them supported first class makes a lot of common tasks quicker to get through.

No, you're right. The canonical way to store a map in capnp is to use a List<Pair<key, value>>. Personally it doesn't bother me that much (mostly from an aesthetic standpoint) because it is not meant as an in-memory, working data structure.

What does bother me is that in the above generic type, "key" and "value" must be pointer types (i.e. non-scalar). The reasons are not unreasonable but it's definitely obnoxious in practice.

HDF5 is quite bad and corruption prone.

Parquet is gaining a lot of traction lately for this use case.

Reply on news.ycombinator.com