NoProto: Flexible, Fast and Compact Serialization with RPC
github.com
github.com
For example, you can certainly compile Avro into Go source files [0]. You can even compile Avro loaded schemas _during runtime_ into Python bytecode, since Python is interpreted [1]. This even works if you have the _wrong schema document_ for the message (you'll just get the subset of fields which are accurately described), because of Avro's schema compatibility rules.
Likewise, you can deserialize arbitrary protobuf messages during runtime without a compilation step, if you have a description for the message schema. The Python protobuf library has had a "ParseMessage" API forever, and protoreflect [2] exists for Go. (In case it's not obvious, I mostly work in Python and Go but I am completely certain analogues exist in other major languages).
There is a very big and important difference between a protocol and the implementation of a protocol. I think this README's author is not clear on that difference, which shows up in other claims ("Deserialization is incrimental", for example) too.
---
[0] https://github.com/actgardner/gogen-avro
[1] https://github.com/spenczar/avroc
[2] https://pkg.go.dev/google.golang.org/protobuf/reflect/protor...
gRPC has a reflection API so that you can dynamically download the schema for messages. It's a lot like Swagger only it works better with less effort.
> all the buffers/data is trusted
This part is also wrong I think. Cap'n'proto is fine to use with untrusted data AFAIK.
> Thus, Cap’n Proto checks the structural integrity of the message just like any other serialization protocol would. And, just like any other protocol, it is up to the app to check the validity of the content.
Likely the author is referring to this section of the home page:
> As of this writing, Cap’n Proto has not undergone a security review, therefore we suggest caution when handling messages from untrusted sources
Which seems more like a disclaimer than "you can't do this" (security is on by default as compared with FlatBuffers where it's opt-in).
But yeah all the stuff about untrusted data is BS. NoProto's readme lists "Safely handle untrusted data" as an advantage vs. every other format it compares against, including Protobuf and JSON. What? Every serious serialization format supports untrusted data.
NoProto also claims to be zero-copy, which would seem to subject it to the same issues. I couldn't find anything in the docs explaining how they avoid it.
I hope he’s also looking at ASN.1 PER for more “don’t write anything the reader already knows” ideas in the vein of Avro.
Without OIDs or UUIDs for versions of types, most of these formats seem equally prone to treating invalid messages as unpredictable gibberish or erroring out on an invalid size or offset.
The only truly fully featured, well-maintained open source implementation for C/C++ seems to be asn1c from Lev Walkin and the dev experience is a world away from what you get using something like Avro, CapnProto or Protobufs.
Really?
Broadly: I'm sorta curious about the details here, but I'll have to read more later. It seems to be claiming the world without a whole lot of evidence, and some incorrect claims, while being implemented as a fairly simple (often good!) serialized form which is also quite large (less good!). Also I have no idea why it's so obsessed with sort-ability. It's not like the encoded data as a whole can be meaningfully sorted, and per-value that sounds like "is big-endian"...
E.g. this is the serialized form of a pre-defined struct with a single field, like `{"age":20}`: https://docs.rs/no_proto/0.9.60/no_proto/format/index.html
// [0, 0, 0, 0, 0, 6, 0, 0, 0, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20]
// [ root ptr, vtable, data]
which is pretty far from protocol buffers': https://developers.google.com/protocol-buffers/docs/encoding 08 96 20
^-- 01 in sample
This sort of waste in favor of simplicity seems to be spread throughout the encoding. I'll happily admit that I think NoProto's format is the easier of the two to read and understand by hand, but that's not generally why we choose binary encodings. It's possible the simplicity gives it a performance edge, but I strongly suspect that, in practice, that's influenced far more by the various implementations' internal details than the encoding itself.I'd love for some team to join that with better performance as it seems like it's possible from what this project has done.