Hacker News Reader

> If you're happy with Go's "everything is a unicode string" approach then you should be happy to just treat everything as unicode. Don't handle the decode errors - if someone sends some data to your stdin that's not in the correct encoding, too bad.

Go's approach is transparently passing data through. In Python 3 your process crashes if you do that.

In my book crashing is better than silently corrupting your data, which is what you'd get if you concatenated two files with different encodings the Go way.

The idea that data corruption happens is wrong though. There is no data corruption, at least not less than in Python 2 now that we have surrogate escape printing support in some cases.

The only way to avoid data corruption is to be explicit about encodings which in 2.x is easy and in 3.x is almost as easy except if you want to work with stdout/stdin.

I totally agree. The transparency of 2.x & Go wins. Python 3.x is a disaster for me, a Unix user in Japan.

Could we all agree to not use the word "unicode" when talking about encoding (ie. how code points are serialized to bytes)? Unicode (ie. the standard set forth by the unicode consortium) has nothing to say about encoding.

I think what you mean is "Go's 'everything is a UTF-8 string' approach", but I'm not familiar enough with Go's internal encoding to know.

For instance, you mention "Don't handle the decode errors", but I can only assume by that you mean UTF-8's decode errors, since UTF-8 has the possibility of having encoding errors, where things like UTF-32 do not. They're both Unicode encodings, so it makes no sense to say it's "Unicode"'s decode errors.

I think the author of this article falls into the same trap all over the place. He uses the word "Unicode" to refer to an encoding all over the place. Until he is able to distinguish the difference between the Unicode standard and all its various encoding methods, there's not much point in reading his article.

(And no, Microsoft's braindead decision to use the word "Unicode" to mean "UCS-2", and later ret-conning it to mean "UTF-16", doesn't count. Don't perpetuate the stupid.)

"Unicode (ie. the standard set forth by the unicode consortium) has nothing to say about encoding."

Yes it does. The UTF-8/16/32 encodings are defined in the Unicode standard (chapter 3.9, page 89), they're not some kind of aftermarket add-on.

"UTF-8 has the possibility of having encoding errors, where things like UTF-32 do not."

Also not true. The set of valid Unicode code points is the same regardless of the encoding (0-0xd7ff and 0xe000-0x10ffff); anything outside those two ranges is just as invalid in UTF-32 as its UTF-8 encoded equivalent would be. A stream containing nothing but 0xff bytes, for example, would be illegal regardless of which UTF you tried to interpret it as.

> Yes it does. The UTF-8/16/32 encodings are defined in the Unicode standard (chapter 3.9, page 89), they're not some kind of aftermarket add-on.

Fair enough, but would you not agree that the word is overloaded? The fact that a few encodings are listed in the Unicode standard doesn't make it ok to wave your hand and call it all "Unicode" in essays like these where it very much matters what actual encoding you're talking about. If you mean UTF-8, stay UTF-8, etc.

> If you're happy with Go's "everything is a unicode string" approach then you should be happy to just treat everything as unicode.

That's actually not really Go's approach. In Go, strings do not have encodings attached to them.

Source files are defined to be UTF-8 (by the compiler), so string literals are always unicode. That's not quite the same thing as saying that the "string" type in Go is always Unicode (it's not). And when you're dealing with a byte slice ([]byte), you cannot make any assumptions about the encoding.

It took a bit to wrap my head around this when I first read about it[0], but now that I think about it, I think it's the right way to go[1].

[0] http://blog.golang.org/strings

[1] And for what it's worth, Go and UTF-8 were designed by (some of) the same people, so one would hope they'd get it right!

You're right, I was lazily responding to the article on its own terms rather than engaging properly with go's string handling.

I think the approach of keeping strings encoded has promise but it would need to be supported by a stronger type system than go's. When you're carrying around two encoded byte arrays it's really important that you know what their encodings are and don't try and e.g. concatenate them. Ruby can do this right because it can give you a runtime type error, but that's not acceptable in a compiled language. So you need to distinguish between byte arrays with statically known encodings, byte arrays with dynamically known encodings and byte arrays with unknown encodings. And you should really e.g. disallow slicing a byte array that represents a string, so that you don't cut a character in half.

I know that one of the UTF-8 guys worked on go and I'm sure go will work well when everything's in UTF-8. But all languages work well when everything's in UTF-8; if anything this makes me more worried that go's authors won't give proper support to those who have to work with strings in non-UTF8 encodings. (By contrast one of the reasons Ruby's support is good is that the author is Japanese and therefore pretty much has to work with strings in multiple non-unicode encodings, because of han unification)

> So you need to distinguish between byte arrays with statically known encodings, byte arrays with dynamically known encodings and byte arrays with unknown encodings.

I don't buy this. You've just introduced some rather large complexity into the types of byte strings just for the sake of handling non-UTF8 cases, which seem to be getting less common than they used to be.

Rust is taking a similar approach to Go (except that their string type must be valid UTF8). You can see a recent debate here: https://mail.mozilla.org/pipermail/rust-dev/2014-May/009725....

If you don't care about handling non-UTF8 cases, you can use pretty much any language (python3 included - the issues the OP is complaining about are when you have filenames in a different encoding from your terminal or the like), write the obvious thing, and it will work fine.

For many use cases that's good enough. But the cases where languages are different, the cases where it gets interesting, are when that isn't enough. (And it won't be enough if you want to sell your software in Japan, for example).

There's actually a bit more to what Rust does: there is a well-known community library called rust-encoding that adds new string types that support various encodings. You can use this library if you need to support other encodings. The standard library supports only UTF-8, but it's simple enough to abstract over strings in multiple encodings if you need to (thanks to generics).

I like this approach: it allows simplicity in the common case, for software that only needs to work in UTF-8, while allowing support for arbitrary other encodings. "Easy things should be easy, and hard things should be possible."

> It took a bit to wrap my head around this when I first read about it[0], but now that I think about it, I think it's the right way to go[1].

I don't actually think it's the right way to go: I'd prefer if strings enforced that they were UTF-8. That saves a lot of error checking by clients that consume it and want to be defensive. It's most efficient to have the type system do the work.

So how do you then deal with problem in the original blog post: a filename that is a non-UTF8 byte string? Surrogates? A separate byte string type?

Use a separate byte string type. The problem is that paths are not UTF-8. That's a fixable problem: just let the type system know about it.