Comment by jchw | Hacker News Reader

Comment by jchw | Hacker News Reader

Parent Full thread

C structs are a terrible serialization format, since they are not a serialization format at all. Nothing guarantees that you will get consistent struct behavior on the same machine, but also, it only really solves the problem for C. For everything else, you have to duplicate the C structure exactly, including however it may vary per architecture (e.g. due to alignment.)

And OK fine. It's not that bad, most C ABIs are able to work around this reasonably OK (not all of them but sure, let's just call it a skill issue.) But then what do you do when you want anything more complicated than a completely fixed-size type? Like for example... a string. Or an array. Now we can't just use a struct, a single request will need to be split into multiple different structures at the bare minimum.

And plus, there's no real reason to limit this all to the same machine. Tunneling UNIX domain sockets over the network is perfectly reasonable behavior and most* SSH implementations these days support this. So I think scoping the interoperability to "same machine" is unnecessarily limiting, especially when it's not actually hard to write consistent de/serialization in any language.

* At least the ones I can think of, like OpenSSH[1], Go's x/crypto/ssh[2], and libssh2[3].

[1]: https://www.openssh.com/txt/release-6.7

[2]: https://pkg.go.dev/golang.org/x/crypto/ssh#Client.ListenUnix

[3]: https://github.com/libssh2/libssh2/pull/945

BTW, Lustre's RPC system's serialization is very much based on C structs. It's receiver-makes-right to deal with endianness, for example. It's a pain, but it's also fast.

Making an RPC serialization system that is zero-overhead i.e. can use the same format on the wire as it does on disk is not a terrible idea. Capnp is the serialization format that I've been suggesting as a potential candidate and it is basically just taking the idea of C structs, dumping it into it's own schema language, and adding the bare minimum to get some protobuf-like semantics.

Well, Lustre RPC doesn't use on-disk data structures on the wire, though that is indeed an interesting idea.

In Lustre RPC _control_ messages go over one channel and they're all a C structure(s) with sender encoding hints so the receiver can make it right, and any variable-length payloads go in separate chunks trailing the C structures.

Whereas bulk _data_ is done with RDMA, and there's no C structures in sight for that.

Capnp sounds about right for encoding rules. The way I'd do it:

  - target 64-bit architectures
    (32-bit senders have to do work
     to encode, but 64-bit senders
     don't)
  - assume C-style struct packing
    rules in the host language
    (but not #pragma packed)
  - use an arena allocator
  - transmit {archflags, base pointer, data}
  - receiver makes right:
     - swab if necessary
     - fix interior pointers
     - fail if there are pointers
       to anything outside the
       received data
     - convert to 32-bit if the
       receiver is 32-bit

(That's roughly what Lustre RPC does.)

As for syntax, I'd build an "ASN.2" that has a syntax that's parseable with LALR(1), dammit, and which is more like what today's devs are used to, but which is otherwise 100% equivalent to ASN.1.

Out of curiosity, why not use offsets instead of pointers? That's what capnp does. I assume offset calculation is going to be efficient on most platforms. This removes the need for fixing up pointers; instead you just need to check bounds.

It's more work for the sender, but the receiver still has to do the same amount of work as before to get back to actual pointers. So it seems like pointless work.

Having actual interior pointers means not having to deal with pointers as offsets when using these objects. Now the programming language could hide those details, but that means knowing or keeping track of the root object whenever traversing those interior pointers, which could be annoying, or else encoding an offset to the root and an offset to the pointed-to-item, which would be ok, and then the programming language can totally hide the fact that interior pointers are offset pairs.

I've a feeling that fixing up pointers is the more interoperable approach, but it's true that it does more memory writes. In any case all interior pointers have to be validated on receiving -- I don't see how to avoid that (bummer).

What a fun sub-thread.

Note within the domain of this problem was the point. Which means on the same machine, with the same architecture and both ends being C which is what the init system is written in.

You are adding more problems that don't exist to the specification.

As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.

> You are adding more problems that don't exist to the specification.

D-Bus does in fact already have support for remoting, and like I said, you can tunnel it today. I'm only suggesting it because I have in fact tunneled D-Bus over the network to call into systemd specifically, already!

> As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.

OK. So... waste an entire page of memory for each string. And then we avoid all of that parsing, but the resulting code is horribly error-prone. And then it still doesn't work if you actually want really large strings, and it also doesn't do much to answer arrays of other things like structures.

Can you maybe see why this is compelling to virtually nobody?

Even being run on the same machine doesn't guarantee two independent processes agree on C struct layout compiled from the same source. For one, you could have something as simple as one compiled for 32bit, one 64, but even then compiler flags can impact struct layout.

> As for strings, just shove a char[4096] in there.

For the love of God, use a proper slice/fat pointer, please.

Switching over to slices eliminates 90%+ of the issues with using C. Carrying around the base and the length eliminates a huge number of the overrun issues (especially if you don't store them consecutively).

Splitting the base and the offset gives a huge amount of semantic information and makes serialization vastly easier.

Broadly, I agree with you. C strings were a mistake. The standard library is full of broken shit that nobody should use, and worse, due to the myriad of slightly different "safe" string library functions, (a completely different subset of which is supported on any given platform,) which all have different edge cases, many people are over-confident that their C string code is actually correct. But is it? Is it checking errors? Does your function ensure that the destination buffer is null-terminated when it fails? Are you sure you don't have any off-by-one issues anywhere?

Correct as you may be though the argument here is that you should just write raw structs into Unix sockets. In this case you can't really use pointers. So, realistically, no slices either. In this context a fixed-size buffer is quite literally the only sensible thing you can do, but also, I think it's a great demonstration of why you absolutely shouldn't do this.

That said, if we're willing to get rid of the constraint of using only one plain C struct, you could use offsets instead of pointers. Allocate some contiguous chunk of memory for your entire request, place struct/strings/etc. in it, and use relative offsets. Then on the receiver side you just need some fairly basic validation checks to ensure none of the offsets go out of bounds. But at that point, you've basically invented part of Cap'n'proto, which begs the question... Why not just use something like that instead. It's pretty much the entire reason they were invented.

Oh well. Unfortunately the unforced errors of D-Bus seem like they will lead to an overcorrection in the other direction, turning the core of our operating system into something that I suspect nobody will love in the long term.

> But at that point, you've basically invented part of Cap'n'proto

The only problem I have with Cap'n Proto is that the description is external to the serialization. Ideally I'd like the binary format to have a small description of what it is at the message head so that people can process messages from future versions.

ie. Something like: "Hmm, I recognize your MSG1 hash/UUID/descriptor so I can do a fast path and just map you directly into memory and grab field FD. Erm, I don't recognize MSG2, so I need to read the description and figure out if it even has field FD and then where FD is in the message."

I thought about this for a bit. I think largely to do things with messages you don't know about is probably a bad idea in general; writing code that works this way is bound to create a lot of trouble in the future, and it's hard to always reason about from every PoV. However, there are some use cases where dealing with types not known at compile-time is useful, obviously debugging tools. In that case I think the right thing to do is just have a way to look up schemas based on some sort of identity. Cap'n'proto is not necessarily the greatest here: It relies on a randomly-generated 64-bit file identifier. I would prefer a URL or perhaps a UUID instead. Either way, carrying a tiny bit of identity information means that the relatively-niche users who need to introspect an unknown message don't cause everyone else to need to pay up-front for describability, and those users that do need introspection can get the entire schema rather than just whatever is described in the serialized form.

It's better to design APIs to be extensible in ways that doesn't require dynamic introspection. It's always possible to have a "generic" header message that contains a more specific message inside of it, so that some consumers of an API can operate on messages even when they contain some data that they don't understand, but I think this still warrants some care to make sure it's definitely the right API design. Maybe in the future you'll come to the conclusion it would actually be better if consumers don't even try to process things they're not aware of as the semantics they implement may some day be wrong for a new type of message.

> I think largely to do things with messages you don't know about is probably a bad idea in general

Versioning, at the least, is extremely difficult without this.

Look at the Vulkan API for an example of what they have to do in C to manage this. They have both an sType tag and a pNext extension pointer in order for past APIs to be able to consume future versions.

But how do you know that the field called "FD" is meaningful if the message is a totally different schema than the one you were expecting?

In general there's very little you can really do with a dynamic schema. Perhaps you can convert the message to JSON or something. But if your code doesn't understand the schema it received, then it can't possibly understand what the fields mean...

Reply on news.ycombinator.com