Hacker News Reader

Hacker News

Comment by nimish | Hacker News Reader

Parent Full thread

nimish

1. Protobuf 2. Avro 3. Thrift 4. MsgPack 5. CORBA 6. ASN.1 7. Cap'n Proto 8. FlatBuffer

+ whatever internal stuff big software companies have cooked up etc.

What was so special about your use-case that demanded a totally new standard?

I hate to bring up that xkcd but it's actually relevant here.

Is it the higher-level semantics on top that allow abstraction over the underlying serialization format?

The "caching" doesn't seem to be that big of a win where network latency is high and some of the other formats can be directly mmapped, but it looks intriguing however it seems like something that could be added in a versioned binary format that some of the others provide.

View on HN

Protobuf - static schema, non self describing Avro - not self describing MsgPack - limited data types (no URLs, Dates, etc) etc.

Go find a format that offers everything transit does, and when you don't find a perfect match for all the goals, you'll understand why this library was created.

Cross-platform (without writing in C), self describing, schema-less, extensible, support for caching, etc.

Avro is self describing [1]. It embeds the schema of whatever it is describing inside the format so a reader can handle arbitrary avro serialized files. It is also cross platform. I'm not sure if avro maps can cache keys, but if you use a record type the keys are not stored in the data. As far as I know though, avro cannot handle custom types, so those will need customized importers and exporters.

[1]: http://avro.apache.org/docs/current/

Why is self-description a necessity here?

Gotta differentiate it from the other stuff somehow.

That is a pretty potshot statement. Please read the objectives section: http://en.wikipedia.org/wiki/Self-documenting

"minimize the effort required to maintain or extend legacy systems" and "reduce the need for users and developers of a system to consult secondary documentation sources" are fitting here.

JSON does all the things Transit does, with fewer types. (And less stupidity)

Should rename it to Enterprise JSON, because it's JSON with more complexity for those architects who don't realize you can easily store a date as a int, or a URL as a string. (Or cache ANY document)

Seriously... why does a document format need support for caching? It would seem to me that a document format should be agnostic to whether it has been cached or not.

Also, why does a document care what language writes it? I don't understand how a document couldn't be cross platform, like maybe if you're using 36-bit words or some fuckery, but most people these days store documents using 8 bit words. Does anyone seriously have issues with JSON on a PDP-10?

> JSON does all the things Transit does, with fewer types. (And less stupidity)

I really hate comments like this.

These guys took the time to show the world this thing they created to fill a need they had, and this comment takes a dump on it without its author first getting any experience using the system. As though the author understands Transit's purpose better than Transit's authors do.

Hey, I get it: Transit /does/ (at first glance) seem largely redundant with all the other serialization libraries out there. But before we assume that its authors spent all this time on their project because they're "stupid", it behooves us to try to understand their motivations.

In the end they're not hurting anyone by releasing this thing they built. If it's bad, you don't have to use it. There's no need to be mean or get upset.

I'm not upset, I'm fine with other people using it. I still think the format is stupid. For the same reasons I think XML is stupid, and no I don't have to use XML, nor do I.

Also, my comment isn't hurting anyone, if you don't like it you don't have to read it, there's no reason to hate :)

No where in my post did I say Rich Hickey is stupid, he has some very great ideas I just don't think this is one of them.

Your comment is hurting people. It hurts the developers, based on at best subjective and at worst ignorant evidence. And it lowers the quality of discussion because people end up having to address your culturally poor behavior rather than the topic at hand.

Being mindful isn't hard.

> if you don't like it you don't have to read it How do we know we don't like it until we've read it?

I upvoted your comment because you make some valid points, e.g. caching should be kept orthogonal to the document format, which needs to be kept simple above almost all else.

I upvoted only after some hesitation, though, because of the unnecessary snark about stupidity. I think that's what you were downvoted for.

Do 1,2,3,4,5,6,7,8 have good, performant implementations in JavaScript - a programming language that many services have to communicate with today?

If the answer is to this question is no, the answer to whether or not a new format is necessary is not automatically yes. Certainly for at least one of those formats, a reasonably performant JS implementation could be created.

We assessed what prior work had been done and found these attempts to ultimately be unsatisfactory performance-wise for the breadth of JavaScript clients we would like to reach.

We have also found that none of the common serialization formats perform acceptably in javascript except JSON (and that's huge on the wire).

As an aside, transit seems dramatically faster in v8 than in firefox, at least in the versons of browsers I'm using, despite the fact that JSON.parse and hydrate are faster in firefox. Has it been specially optimised for v8?

We did not specifically optimize of for V8, the optimizations present resulted in a performance win in all browsers. Firefox simply does not deliver the same performance as V8 or JavaScriptCore for this kind of work. Still I think Transit is plenty acceptable under Firefox for many typical JavaScript programs. Hopefully the existence and usage of Transit will encourage Firefox to further improve their JavaScript performance profile.

I would love to see this data--it would help me greatly when picking a format that needs to work.

http://jsperf.com/json-bson-msgpack/2

Look at JSPerf for other serialization formats vs. JSON. They all look pretty much like this. Compare these results to http://jsperf.com/json-vs-transit/2 where in some cases we beat JSON.

Possibly but in many ways it's a lot easier to pick and write a fast implementation using e.g. typed arrays than it is to convince the world to use your special format

Do typed arrays work in the last 14 years of browser technology or JavaScript environments?

It would be very nice if y'all gave a detailed rationale mentioning things like this over "we made a cool new format"

I still am not seeing why the other formats fail, especially with the very limited compression that you have baked into the spec (!)

Where are the benchmarks on ie6 era browsers? And why should I let ie6 era perf direct my future data format design?

Reply on news.ycombinator.com