Well there sort of is but people don't tend to know or use it. If it's within the same machine and architecture, which should be the case for an init system, then a fixed size struct can be written and read trivially.
And OK fine. It's not that bad, most C ABIs are able to work around this reasonably OK (not all of them but sure, let's just call it a skill issue.) But then what do you do when you want anything more complicated than a completely fixed-size type? Like for example... a string. Or an array. Now we can't just use a struct, a single request will need to be split into multiple different structures at the bare minimum.
And plus, there's no real reason to limit this all to the same machine. Tunneling UNIX domain sockets over the network is perfectly reasonable behavior and most* SSH implementations these days support this. So I think scoping the interoperability to "same machine" is unnecessarily limiting, especially when it's not actually hard to write consistent de/serialization in any language.
* At least the ones I can think of, like OpenSSH[1], Go's x/crypto/ssh[2], and libssh2[3].
[1]: https://www.openssh.com/txt/release-6.7
[2]: https://pkg.go.dev/golang.org/x/crypto/ssh#Client.ListenUnix
You are adding more problems that don't exist to the specification.
As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.
D-Bus does in fact already have support for remoting, and like I said, you can tunnel it today. I'm only suggesting it because I have in fact tunneled D-Bus over the network to call into systemd specifically, already!
> As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.
OK. So... waste an entire page of memory for each string. And then we avoid all of that parsing, but the resulting code is horribly error-prone. And then it still doesn't work if you actually want really large strings, and it also doesn't do much to answer arrays of other things like structures.
Can you maybe see why this is compelling to virtually nobody?
For the love of God, use a proper slice/fat pointer, please.
Switching over to slices eliminates 90%+ of the issues with using C. Carrying around the base and the length eliminates a huge number of the overrun issues (especially if you don't store them consecutively).
Splitting the base and the offset gives a huge amount of semantic information and makes serialization vastly easier.
Correct as you may be though the argument here is that you should just write raw structs into Unix sockets. In this case you can't really use pointers. So, realistically, no slices either. In this context a fixed-size buffer is quite literally the only sensible thing you can do, but also, I think it's a great demonstration of why you absolutely shouldn't do this.
That said, if we're willing to get rid of the constraint of using only one plain C struct, you could use offsets instead of pointers. Allocate some contiguous chunk of memory for your entire request, place struct/strings/etc. in it, and use relative offsets. Then on the receiver side you just need some fairly basic validation checks to ensure none of the offsets go out of bounds. But at that point, you've basically invented part of Cap'n'proto, which begs the question... Why not just use something like that instead. It's pretty much the entire reason they were invented.
Oh well. Unfortunately the unforced errors of D-Bus seem like they will lead to an overcorrection in the other direction, turning the core of our operating system into something that I suspect nobody will love in the long term.
The only problem I have with Cap'n Proto is that the description is external to the serialization. Ideally I'd like the binary format to have a small description of what it is at the message head so that people can process messages from future versions.
ie. Something like: "Hmm, I recognize your MSG1 hash/UUID/descriptor so I can do a fast path and just map you directly into memory and grab field FD. Erm, I don't recognize MSG2, so I need to read the description and figure out if it even has field FD and then where FD is in the message."
In general there's very little you can really do with a dynamic schema. Perhaps you can convert the message to JSON or something. But if your code doesn't understand the schema it received, then it can't possibly understand what the fields mean...
It's better to design APIs to be extensible in ways that doesn't require dynamic introspection. It's always possible to have a "generic" header message that contains a more specific message inside of it, so that some consumers of an API can operate on messages even when they contain some data that they don't understand, but I think this still warrants some care to make sure it's definitely the right API design. Maybe in the future you'll come to the conclusion it would actually be better if consumers don't even try to process things they're not aware of as the semantics they implement may some day be wrong for a new type of message.
Versioning, at the least, is extremely difficult without this.
Look at the Vulkan API for an example of what they have to do in C to manage this. They have both an sType tag and a pNext extension pointer in order for past APIs to be able to consume future versions.
In Lustre RPC _control_ messages go over one channel and they're all a C structure(s) with sender encoding hints so the receiver can make it right, and any variable-length payloads go in separate chunks trailing the C structures.
Whereas bulk _data_ is done with RDMA, and there's no C structures in sight for that.
Capnp sounds about right for encoding rules. The way I'd do it:
- target 64-bit architectures
(32-bit senders have to do work
to encode, but 64-bit senders
don't)
- assume C-style struct packing
rules in the host language
(but not #pragma packed)
- use an arena allocator
- transmit {archflags, base pointer, data}
- receiver makes right:
- swab if necessary
- fix interior pointers
- fail if there are pointers
to anything outside the
received data
- convert to 32-bit if the
receiver is 32-bit
As for syntax, I'd build an "ASN.2" that has a syntax that's parseable with LALR(1), dammit, and which is more like what today's devs are used to, but which is otherwise 100% equivalent to ASN.1.
Having actual interior pointers means not having to deal with pointers as offsets when using these objects. Now the programming language could hide those details, but that means knowing or keeping track of the root object whenever traversing those interior pointers, which could be annoying, or else encoding an offset to the root and an offset to the pointed-to-item, which would be ok, and then the programming language can totally hide the fact that interior pointers are offset pairs.
I've a feeling that fixing up pointers is the more interoperable approach, but it's true that it does more memory writes. In any case all interior pointers have to be validated on receiving -- I don't see how to avoid that (bummer).
What a fun sub-thread.