Edit: It occurred to me after posting this that by "length + data" you might mean actually storing the length
with the string data. If so, you're right, that's stupid. The
correct solution (well, it's better than C/Pascal strings, anyway) is to store
just the string data in the string proper, and use references that consist of pointer+length or pointer+pointer pairs instead of single char-pointers.
> It's easy to break a string with delimiters into the individual pieces in place, just by writing nulls over the separating characters
This assumes you're free to modify the input. If you don't want/are unable to modify the input, you're forced to allocate memory to hold the output; you've totally unnecessarily doubled the space requirement. It also assumes that there is a separating character to overwrite, which may not be the case, causing the same problem.
> just by writing nulls over the separating characters
> This can't be done with some other string representations like length + data
Obviously. It is, however, perfectly possible to tokenize a string into a vector of length+data pairs without mangling the input.
> When one null terminated string is a suffix of another (and ideally both are treated as immutable), then they can share storage.
And only when one is a suffix of another. If you want to share storage between strings between which this relationship does not hold, you must again allocate extra memory, and again this is not necessary if you use a data+length representation.
> Catenating null terminating strings is efficient if you keep a tail pointer
In other words, if you use a pointer+length or arithmetically equivalent representation—except that you don't even bother binding the data into a coherent data structure, and instead force any code without access to your local tail-pointer variable to derive the length/end of the string on its own.
> For one thing, how wide should be the length?
Wide enough to represent the length of the longest possible string. Generally, this is less than or equal to the length of a pointer.
> If you make it two bytes today and store binary data somewhere, it will be incompatible with tomorrow's four byte length
> Null terminated strings can be blasted over a serial line or network, or written to disks, as they are
You are making the incorrect assumption that the most appropriate on-disk/serialization representation is necessarily the same as the most appropriate in-memory representation.
> And then there is endiannness
1. See above.
2. You're right that "this isn't true of wide character null term'd strings," so unless you and everyone you communicate with are still living in 1985, NTBSes don't even have that advantage.
> You cannot pass them between address spaces or share them without marshaling to some flat form and back
You can't do this with NTBSes, either, unless you have a way of translating the address of the string across address spaces, in which case you could just as well pass a pointer+length as you could pass a pointer.
NTBSes are a perfectly fine representation if: your string is known at compile time; your string will never be modified; your string will only be scanned through from front to back; and your string does not contain a null character. From a technical standpoint, the only reason to bother with them in the first place is that you have extremely tight memory requirements and can't even afford the extra byte or three needed for a sane string-reference representation.
If any of these conditions does not hold, NTBSes are a Terrible, Horrible, No Good, Very Bad representation. All they accomplish in practice is to make strlen a linear-time operation, defeat a great number of data-sharing opportunities, and generally plague the world with buffer overruns.