> 4. contains no null bytes, so it can fit in any normal C string
This is false; the encoding of codepoint 0 is 0x00 per the standard; modified utf-8[1] makes 0 a special case with an over-long encoding: 0xC0 0x80.
This is false; the encoding of codepoint 0 is 0x00 per the standard; modified utf-8[1] makes 0 a special case with an over-long encoding: 0xC0 0x80.
And, if you really need to represent codepoint 0 in strings, you can use Java's Modified UTF-8, where codepoint 0 is represented by the byte sequence 0xC0, 0x80. (This isn't valid UTF-8 because in straight UTF-8, every codepoint must be represented by its shortest possible representation.)
No it doesn't, unless you are saying that one should treat it like that. But null termination is as dangerous[1,2] with UTF-8 as it is with ASCII and should be avoided as much as possible anyway. Also, ASCII doesn't mandate that \0 is end-of-string, that's just a "convention" from C.
(Did you notice that my original comment actually included the exact modified UTF-8 link you provided?)
[1]: http://cwe.mitre.org/data/definitions/170.html [2]: http://projects.webappsec.org/w/page/13246949/Null%20Byte%20...
MODULE_LICENSE("GPL\0for files in the \"GPL\"
directory; for others, only LICENSE file applies");
[1] https://en.wikipedia.org/wiki/Loadable_kernel_module#Linuxan...
Well, yes. That's the point: 0x00 is only ever used to encode codepoint 0. It never shows up anywhere else. That's precisely what the text you replied to means.
even the C guys have moved on though, as Go (co-designed by Ken Thompson) illustrates.
This is indeed a tradeoff. The only alternatives are to either store the length of the string explicitly somewhere (the Pascal solution is to prepend the string with a fixed-size length parameter, which can be as awkward as it sounds when you get to really long strings) or to do something nasty with the string's actual contents, such as saying that the last byte of a string has its high bit set.
I think the C solution is the most reasonable when storage is really tight, and more flexible than the Pascal method in general, but I agree that it's potentially dangerous and it's annoying to have a byte which you can never represent in a string.
Why? C strings and Pascal strings can store exactly the same contents, except Pascal strings can store a literal \0, and are faster to manipulate (in many ways), at the cost of (sizeof(word) - 1) extra bytes.
C-strings are useful in extremely constrained environments when the extra few bytes of the length prefix vs the trailing \0 byte is too much to pay, but are essentially just a security risk in any other situation.