It's a huge usability fail on the part of Python that it's not obvious and easily memorable how to do this.
If you think about it, that's the only way that makes sense; if you wanted to go from, say, UTF8 to UTF16, you'd need to have an intermediate step that was abstract and formatless. Similarly, when you're operating on text, you don't want it to have a particular format; the format should only exist at the edges, where you take text in and output it. The format is a way to communicate with other programs, not components within your program.
My advice would be, if that doesn't stick in your head (totally legitimate), write down a code example and stick it in a personal wiki or something. I find it really helps me to understand something once and more or less copy paste it forever.
This is a key reason why people hated Python 3 so much IMO.
For context I’m a huge advocate of Python 3, except for this (and lambda functions, but that’s another whine session).
https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
A simplistic summary is that you always decode to unicode and always encode to bytes.
Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage.
It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.
The problem is that encode and decode are directional constructs that change depending on context.
That’s a poor UI in Python.
It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such.
Taking a stab at it; This looks simple, and runes are great (similar to Python):
https://golangdocs.com/strings-in-golang
But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say.
This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays:
I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree.
If I'm not understanding properly I'm happy to be corrected.
byte[] unicodeBytes = Encoding.UTF8.GetBytes(inputString);
// Perform the conversion from one encoding to the other.
byte[] asciiBytes = Encoding.Convert(Encoding.UTF8, Encoding.ASCII, unicodeBytes);
string outputString = Encoding.ASCII.GetString(asciiBytes);
utf8_bytes: bytes = bytes(input_string, encoding="utf-8")
# Perform the conversion from one encoding to the other.
unicode_string: str = str(utf8_bytes, encoding="utf-8")
ascii_bytes: bytes = bytes(unicode_string, encoding="ascii")
# The conversion could also be written as:
ascii_bytes: bytes = utf8_bytes.decode("utf-8").encode("ascii")
output_string: str = str(ascii_bytes, encoding="ascii")
Perhaps a convenience method would be nice, something like this, but it somewhat obscures the intermediate decode-to-unicode step:
ascii_bytes: bytes = utf8_bytes.convert(to_encoding="ascii")Isn't it true that encode always means text -> bytes and decode always means bytes -> text?