Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.
Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct.
To call that "deferring the parsing to access time" does not make sense.
(Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.)
Your statement earlier:
> The correct way to handle binary data is to unpack it into the struct byte by byte.
This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof.
On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works".
(Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)
As long as the Cap'n Proto or ProtoBuf data is properly aligned within of your custom file format, or you might end up with unusual slowness: https://blogs.msdn.microsoft.com/oldnewthing/20150116-00/?p=...
(Protobuf, on the other hand, fundamentally doesn't allow for multi-byte loads in the first place since integers use variable-width encoding, so alignment is irrelevant there.)
Basically, a decade ago a student started to work with some friends on an IRC client that integrates with a custom bouncer. Being a prototype, they just used Qt's serialization protocol between them. Over the years the project grew, at some point nokia funded development, it became Kubuntu's default IRC client (but only in the client+bouncer in one binary version, so without all the advantages), nokia was bought and closed and the department sold to BMW, and at some point.
Now, people tried writing third-party clients for this. And this became a minor issue, because the protocol was never documented. In favt, Qt's serialization was used for storing configs on disk, some blobs in the database, and over the network. It was later wrapped in TLS and deflate, and even at some point array-of-struct was turned into struct-of-array for the pattern during initialisation.
Either way, someone tried writing a mobile client for it, decided the protocol was insane, and instead built his own, almost identical client/bouncer system with an Erlang backend and json as protocol. This grew, and became IRCCloud.
Now, other people again tried developing third party clients for quassel. An android client was developed, but development was messy, and over the years, it stalled, because they reverse engineered the protocol, semi-successfully, and at some point didn't have enough time left, and gave up.
People reversed the protocol partially over the years again for pyquassel and quasselc/quasselbots/quassel-irssi.
Around then, another person tried reversing the protocol, and reimplementing it in JS for a webclient, which after a while became quassel-webserver.
Back then there was a lot of talk about replacing the protocol, but it was never done yet.
I, a user of irccloud around then, was annoyed with the costs (still being in high school myself, I couldn't afford the 4$/month, and the free tier wasn't enough), so I out for alternatives, and found quassel. But I hated the looks of Quasseldroid, so I forked it, and started working on the UI, and on features by reversing the protocol yet again. Not having actually programmed anything except for some delphi and VB.NET stuff, a tiny java project and one C# Windows Phone 7 app, my code was the worst, ever. Seriously, it was bad. After a while, discussion came about about turning this code into a PR, so I threw it all away, and rewrote it again, still bad, but it worked. This was merged, I became maintainer of Quasseldroid (because no one else was working on it anymore), and then, around 2015, I reversed the protocol, read the entire source of every implementation, wrote it all down on paper, studied every file format and every quirks and then I rewrote quasseldroid from scratch, in about 3 months, with every feature of the desktop version. I called this The Next Generation of Quasseldroid, jokingly quasseldroid TNG or later quasseldroid-ng.
A few weeks later, a new Android version came out, introducing Doze, and breaking everything about quasseldroid-ng. And so I rewrote it again, and before release, a new Android version broke it all again.
And that basically repeated, until I decided that it can't continue like this, and if we'll do major changes to the protocol, we might actually get a working version for Android that lasts longer than a few weeks in Beta before Google breaks it. So I learnt C++, and started contributing.
And that is basically my view of the story. Reverse engineering a protocol again and again, seeing variations of variations of the same protocol everywhere used, never properly documented. The Qt documentation is entirely different from what Qt actually puts on the wire. Blobs in the database.
But it also means backwards compatibility in the desktop client/core for every version from the past 10 years, and backwards compatibility on Android for every version of the past 6 years.
And now, maybe, I'll be able to help replace this, bit by bit. After the improvements to tge protocol that added major performance benefits, the next part is replacing the bouncer-side config format entirely, so I can properly containerise it.
TL;DR: no matter how good the support for a non-standard binary serialization format in your favourite language is, 10 years down the road people will reuse your protocol in a dozen more languages, and they'll have to reverse the protocol themselves, and will do a semi-good job at it, and because you never thought about backwards compatibility you now have a mess (we had luck because 99% of what we transmitted were key/value maps, and when reading we always used default values if the key didn't exist and ignored unused keys. Sometimes we did serialize structs, basically, though, and those places still cause me headaches today, and require workarounds to add features, e.g. the latest sendermode implementations)
Building upon an existing, well-documented, and relatively sane serialization format (protobuf, capn't proto, message pack, json, heck even bencode for all I care) is usually a good thing, and so is decoupling the messages from the details of an implementation's internals. Language and framework internal serializers (such as Python's pickle or, apparently, Qt's serializer) tend to make it harder to achieve both goals.
The problem with that is that whatever format seems well-documented and relatively sane today might become an obscure, unknown protocol 10 years down the road.
I unfortunately am not in a position to make such strong statements about Cap'n Proto. However, implementations exist in C++, Java, JavaScript, Rust, Go, Python, and a bunch of other languages, so it should at least be much easier to deal with than Qt serialization.
(Disclosure: I'm the author of Cap'n Proto and of the first open source release of Protobuf.)
When was the last time you saw a filename of the form NODE"accountname password"::device:[directory.subdirectory]filename.type;ver?
Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. (And you assume all Windows instances are little-endian, which probably-is-but-may-not-be the case.) You made the decision to optimize for what you consider the common case, but it does not generalize without added translation code to all cases. You may have hidden the translation code behind CapnProto's generated accessors, but CapnProto's structs are translation-free the way AWS Lambda is "serverless".
> To call that "deferring the parsing to access time" does not make sense.
Except you are deferring translation work (like byte reordering) to access time. Either that or you're hiding it in the serialization APIs. Again, it's like "serverless" computing: just because you've hidden it doesn't mean it's gone away.
> Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid.
There's a performance cost, but hopefully you're only doing serialization/deserialization when you intend to hit the disk or wire to read/write into/out of your struct. All in-memory processing happens in whatever endianness and alignment makes your CPU and compiler happy.
There's nothing "overly complex and rigid" about understanding a binary format as a bag of bytes and fetching scalar values (including offsets into the data structure) from it accordingly. This is how shit gets done when it comes to portably handling arbitrary binary formats. CapnProto can score a few wins by assuming things about the target CPU/compiler, restricting the binary format to conform to some of those assumptions, and papering over the rest with code hidden behind some of its APIs. But it's not a general solution to the problem of extracting meaning from an arbitrary hunk of bytes that may or may not have come from a CapnProto-conformant application.
Basically all common CPUs are LE.
(And basically all BE CPUs have dedicated instructions for reading LE data. It's true I haven't yet added the inline assembly to use those instructions in Cap'n Proto's reference implementation, but that's only because no one actually cares about these architectures.)
So in basically all real use, there's no machine-instruction-level difference between accessing a field of a capnp struct and accessing a field of a C struct. If the instructions are identical, then how can you say one is "deferred parsing" and the other isn't? What meaning does any such distinction have?
> There's a performance cost, but hopefully you're only doing serialization/deserialization when you intend to hit the disk or wire to read/write into/out of your struct.
Disk is usually cached, meaning it's already in physical memory and you're wasting time making a copy rather than using the data in-place.
Over the network, within a datacenter, bandwidth is basically infinite (in that your CPU probably can't process bytes as fast as your network interface can). Time spent serializing and parsing is very real and wasteful. I've seen servers spending 30% or more of their CPU time parsing protobufs.
Over the long-haul internet, perhaps the CPU time spent parsing/serializing is not as relevant compared to the time spent transmitting. Still, I'd rather spend my CPU cycles elsewhere -- like in a dedicated compression algorithm -- rather than twiddling bytes needlessly in a parser.
> handling arbitrary binary formats
Yes, we all agree that some binary formats can't be handled any other way. But if you're in control of the format you use, then you can design it in a way that doesn't require a "bag of bytes" model, and your code can be much simpler and more adaptable as a result.
This. Optimizing performance on BE is a waste of everyone's time and resources.
(But the status quo on BE is that it does a load followed by a byte swap, which is probably pretty cheap anyway. The compiler might even already know how to optimize that into the appropriate LE-load instruction.)
I have to agree here by experiences past. If the format in question has a chance of being performance sensitive, don't use FSM-based encodings [1]. It is inordinately difficult to optimize parsing these encodings even if you only have to handle tiny subsets, and it still won't be fast. A format like msgpack which prides itself on being very fast may be fast compared to JSON and other ways to express essentially arbitrary structures, but is DEAD SLOW compared to any direct encoding (be it a dedicated encoding you developed in literally a few hours or something like capnproto).
[1] Obviously, considering an encoding more complex than FSM means that you're an idiot and your application will almost certainly have security vulnerabilities related to the format in the future.