Hacker News Reader

Comment by saila | Hacker News Reader

It's not that complicated nor is it a Python specific issue. This is a good article on the topic:

https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

A simplistic summary is that you always decode to unicode and always encode to bytes.

Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage.

It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.

View on HN

It’s not that I don’t understand it.

The problem is that encode and decode are directional constructs that change depending on context.

That’s a poor UI in Python.

Genuinely curious, what would you like it to look like?

golang has a wonderful solution

Would you care to elaborate on what it is and why it's better for the benefit of those of us who don't know Go?

It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such.

Taking a stab at it; This looks simple, and runes are great (similar to Python):

https://golangdocs.com/strings-in-golang

But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say.

This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays:

https://go.dev/blog/strings

I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree.

If I'm not understanding properly I'm happy to be corrected.

I don't have skin in the game, but at least the .Net way is quite explicit, which I like:

    byte[] unicodeBytes = Encoding.UTF8.GetBytes(inputString);

    // Perform the conversion from one encoding to the other.
    byte[] asciiBytes = Encoding.Convert(Encoding.UTF8, Encoding.ASCII, unicodeBytes);
         
    string outputString = Encoding.ASCII.GetString(asciiBytes);

Adapted from https://learn.microsoft.com/en-us/dotnet/api/system.text.enc...

You can do something similar in Python:

    utf8_bytes: bytes = bytes(input_string, encoding="utf-8")
    
    # Perform the conversion from one encoding to the other.
    unicode_string: str = str(utf8_bytes, encoding="utf-8")
    ascii_bytes: bytes = bytes(unicode_string, encoding="ascii")
    
    # The conversion could also be written as:
    ascii_bytes: bytes = utf8_bytes.decode("utf-8").encode("ascii")
    
    output_string: str = str(ascii_bytes, encoding="ascii")

The biggest difference is that the conversion step requires you to explicitly decode the bytes to a unicode string and then encode the unicode string back to bytes rather than providing a convert() method that does this internally.

Perhaps a convenience method would be nice, something like this, but it somewhat obscures the intermediate decode-to-unicode step:

    ascii_bytes: bytes = utf8_bytes.convert(to_encoding="ascii")

Could you elaborate on that?

Isn't it true that encode always means text -> bytes and decode always means bytes -> text?

Reply on news.ycombinator.com