I'd argue that strings and bytes are the same general type, but it's sometimes useful to give well-formed utf8 bytes a different type internally. Rust gets this mostly correct with OsString and String.
The thing I think Rust maybe goofed, or at least made a little complicated, is their weird distinction between a String and a str (and a &str). As a newbie learning the language, I have no idea which one to use, and usually just pick one, try to compile, then if it fails, pick the other one. I'm sure there was a great reason to have two types for the same thing, that I will understand when I know the language better.
If you want to understand more deeply, the Rust Programming Langauge, chapter 4, uses String and &String and &str to talk about ownership and borrowing. Here’s a link to the start of that chapter: https://doc.rust-lang.org/stable/book/ch04-00-understanding-...
Your blog post is practical and clearly explains what to do, when, which is helpful. What's confusing is why Rust has the two types and why the language designers decided it was a good idea to have to convert back and forth between them depending on whether it was going in a struct or being passed as an argument. I suppose the "why" is probably better found in the Rust docs.
As a long-time C++ user, it seems like std::string vs const char* all over again, and we somehow didn't find a better way.
It’s closer to std::string and std::string_view. But yes, in a language with value and reference semantics, when you also care about performance, you just can’t do any better: you need both types. Or at least, if you want the additional correctness guarantees and safety provided by communicating ownership semantics in the type. C gets away with just char * but then you have to read the docs to figure out what you’re allowed to do with it and what your responsibilities are.
A pointer to some memory is not the same thing as a struct that has a pointer to memory, as well as a capacity field and the ability to resize itself.
To give a real example, I once wrote some python scripts to parse serial messages coming off a bus. They'd read the messages, extract some values with regex, and move on.
Unfortunately the bus had some electrical bugs and would intermittently flip random bits with no CRC to correct them. From my point of view, no big deal. If it's in something outside the fields I care about, I won't notice it. If it's flipped something I do care about we have a bad sample to drop or noise the signal processing will deal with. Either way, it's fine. Python on the other hand cared very much. I rewrote everything in C once I got sufficiently annoyed of dealing with it and more importantly explaining to others how they couldn't "simplify" things using the stdlib APIs.
// NB: Must be utf-8!
struct string {
size_t sz;
size_t capacity;
unsigned char *buffer;
};
&String in Rust is roughly like `const struct string *`.str in Rust is just an array of (guaranteed utf-8) unsigned bytes. It does not have a capacity, so it can't be resized. You can't directly construct one (on the stack), because its size is undetermined and Rust doesn't have dynamic-sized stack allocation.
&str, and Box<str>, are pointers to str, along with a size, and are roughly like this C:
// NOTE: Must be utf-8!
struct str_ptr
{
size_t sz;
unsigned char *buffer;
}
The difference between &str and Box<str> is that the latter is an owned pointer to a heap allocation which will be freed when it goes out of scope. &str is unowned and might point anywhere: to a Box<str> on the heap, to a String on the heap, or to read-only static memory.IMO, it's probably easier to first try to understand the difference between `Vec<u8>`, `&[u8]`, and `&Vec<u8>`, because they are slightly less "weird" than the string types: they aren't syntactically special like `str` is[1], and they don't have an implicit requirement to be utf8 that is inexpressible in the type system.
[1]: `str` is syntactically special because it is basically a slice, but isn't written in slice notation.
Rust could have done better in naming, but a definite design goal of the language (for better and worse) is to not make things that are complicated for the compiler appear simple to the user. Which unfortunately results in:
String/str
CString/CStr
OsString/OsStr
Vec<u8>/[u8]
AsRef<str>
Cow<`a, str>Of course if you provide a separate set of functions for treating a string as human readable vs not you can also work with that. Basically len() vs byte_len().
But you can’t concat two human readable strings without ensuring they are of the same encoding. You can’t search a string by bytes if your needle is of a different encoding. You can’t sort without taking encoding and locale preferences into account, etc.
Pretending like you don’t care about encoding doesn’t work as we have seen time and again.
At the language level C historically hasn't offered much support for working with specific character sets and their encodings. With C17 and C23 we get u"...", U"...", u8"...", type char8_t, and similar, but there's still little/no built-in tooling for text processing.
For text processing work with char* whose bytes are some encoding/s of Unicode, e.g. UTF-8, then you an use a C library such as libunistring or ICU.
However the bytes of a char* could instead be an encoding of a non-Unicode character set, e.g. GB2312 encoded as EUC-CN.
So char* is character set and encoding agnostic. And C-the-language doesn't even try to offer you tools for working with different sets and encodings. Instead, you can use a library or write your own code for that purpose.
A number of languages make the same decision, keeping the string type set/encoding agnostic, with libraries taking up the slack.
In Nim, for example, the string type is essentially raw bytes (string literals in .nim sources are UTF-8). If you're doing Unicode text processing then you'd use facilities from the std/unicode module
https://nim-lang.org/docs/unicode.html
Same story with Zig
https://ziglang.org/documentation/0.8.0/std/#std;unicode
Lua too, and you'll probably use a 3rd party library such as luaut8 for working with Unicode/UTF-8
https://github.com/starwing/luautf8
Returning to the matter of pretty.c, since it's just sugar for C, it makes sense (to me) that the string type is just an alias for the set/encoding agnostic char*. It's up to the programmer to know and decide what the bytes represent and choose a library accordingly.
- String data will be properly encoded
- There is one encoding of strings (UTF-8 usually)
- Validation must occur when string data is created
- Truncating a logical codepoint is never acceptable
- You may not do string things to "invalid" bytes
- Proper encoding is the beginning and the end of validation
None of these things are consistently true. It's a useful practice to wrap validated byte sequences in a type which can only be created by validation, and once you're doing that, `Utf8String` and `EmailAddress` are basically the same thing, there's no reason to privilege the encoding in the type system.If it's "human-readable text", then fine, a string is not the same thing as an arbitrary byte array.
But lots of languages don't enforce that definition.