Hacker News Reader

Comment by asveikau | Hacker News Reader

One approach I've seen to writing big endian safe software is to not load integers directly, and apply the same logic regardless of endianness.

Eg. Instead of:

       uint16_t x;
    #ifdef BIG_ENDIAN
       x = swap(*p);
    #else 
       x = *p;
    #endif

You can write:

       uint16_t x;
       unsigned char *q = (unsigned char*)p;
       x = q[0] | (((uint16_t)q[1]) << 8);

(Pardon the rushed job of writing C in an HN form on mobile.)

This exercises the same deserialization from little endian regardless of the hardware representation of integers. This means you are testing how it will act on big endian even when running on little endian.

View on HN

The problem is that it's hard to be certain that you actually did it correctly everywhere if you aren't testing it. Writing endian-neutral code isn't terribly difficult, but you can easily accidentally write endian-dependent code here and there.

This works when your serialization format is designed to be either big or little endian. But for some programs, the serialization format is memcpy the structure to disk. This isn't a good strategy, but unfortunately it can be difficult to undo historical choices.

So the padding, alignment, endianness, all leak. And it sounds like X does something like this: clients expect the endianness of serialized data to match the local machine and the server can optionally account for this.

I've done things like this.

First, the structs need the nonstandard pragmas and attributes to not pad.

Then, all accesses of 16, 32, or 64 bit quantities need to go through a helper function.

Even with that baggage, the syntax of having struct member access is can still feel more convenient than more verbose serialization code.

why not just use POSIX hton{,s,l} ?

https://linux.die.net/man/3/htons

Code like that second one (using shifts) is generally what I do always (dealing with small-endian, big-endian, or PDP-endian data), anyways. Hopefully, the compiler will be able to optimize the code so that it works like the first one. (I don't know if the compiler actually does optimize it or not, though. However, it seems to me that a code like that should be common enough that it would make sense to implement such an optimization.)

Neither of those are best practice; you should use some kind of uint16_from_le() macro.

Yep. Rust's APIs do this right. Eg u32::to_be_bytes() / u32::to_le_bytes() and u32::from_be_bytes() / u32::from_le_bytes().

There's also methods for native byte order, but thats almost never the right thing to use.

I think Go does the same thing. I heard about this approach first from Rob Pike, who would rant about this to anyone who would listen. He said to think of "bytes on disk/network" and "an integer in RAM" as different types that you need to convert between. When you do, the API should specify the endian-ness of the bytes on disk / in the network. He argued you shouldn't do "byte swapping" at all based on endianness. Just parse the int, byte at a time and use the same code on every computer.

The real problem is that C makes it easy to convert between a pointer to an integer and a pointer to some bytes in RAM. Just never do that using pointer casting, and everything works out great.

https://doc.rust-lang.org/std/primitive.u32.html#method.to_b...

We agree. I'm writing it out as an example. In real life you would put it in some kind of helper function. (Or macro.) I'm not suggesting writing it out inline on every access.

In that case I disagree. You can't guarantee that the shift-and-or will be optimized out by the compiler in cases where no swapping is needed so you're making the code potentially slower.

So.. ntohl?

If you put that in a helper function, then kind of.. except ntohl and ntohs deal with big endian. And is a no-op on big endian. And there is no ntohl for 64 bit quantities.

Reply on news.ycombinator.com