This works when your serialization format is designed to be either big or little endian. But for some programs, the serialization format is memcpy the structure to disk. This isn't a good strategy, but unfortunately it can be difficult to undo historical choices.

So the padding, alignment, endianness, all leak. And it sounds like X does something like this: clients expect the endianness of serialized data to match the local machine and the server can optionally account for this.