Hacker News Reader

Comment by fuzzix | Hacker News Reader

> i18n just happens

No it doesn't... unless machine translation is "there" and built into Tcl.

> Every string is internally encoded in utf-8, all the string operations are Unicode-safe, including the regular expression engine. Basically, in Tcl programs, encodings are not a problem - they just work

Really? Perl's Unicode support is pretty much second-to-none, IME, but you still need to know the encoding of your file handles and so on. Once it knows this, Perl will Just Work(TM). How is this handled in Tcl?

View on HN

I don't understand the purpose of your comment. OK so perl is great. This article isn't about Perl...it's about the semi-obscure and commonly underestimated Tcl (which is also a great language).

I don't think fuzzix was trying to divert the discussion onto Perl, but citing it as an example of a language generally agreed to do an excellent job of handling character encodings in which it still isn't true (as the article alleges for Tcl) that "encodings are not a problem - they just work".

The purpose of the comment seems clear enough to me: to point out two claims made by the article that seem implausibly optimistic:

1. That i18n "just happens", which as fuzzix says is surely impossible unless Tcl includes magical machine translation facilities.

2. That "encodings are not a problem - they just work", which would require Tcl to tell by some ingenious means what encoding is used by any given file -- something that Perl, despite being exactly the sort of language that doesn't mind the kind of heuristic complexity it would take to do this well, and being generally acknowledged to do a good job of handling Unicode, punts on).

To state the problem explicitly, internal storage format of strings doesn't mean you can handle all data streams seamlessly. Internal consistency is fine, but ultimately your program will have I/O.

If I open an ISO-8859-15 or UCS-2 file in a Tcl program, how is this handled? Does it seamlessly detect encoding and translate to UTF-8 or is there more to this issue than what's stated in the article? Also, are the strings stored in a canonicalised form? If not, how simple is it to perform normalisation?

I brought up Perl because it's my experience of a language with excellent encoding support and helps make the point that the issue is probably more complex than just how strings are stored and manipulated internally.

This article is from 2006.

Perl's unicode support is excellent now. In 2006 5.10 wasn't out yet, and anything before 5.8.5 had all sorts of odd unicode bugs.

Or: Tcl got there significantly before we did.

That is a fair point, but I'm not trying argue that Perl is somehow better. I am just using it as a reference point. I'm more suggesting the article may be overstating the ease-of-use / capabilities of Tcl.

To be clear, I'm not arguing against using Tcl in favour of Perl or anything else. It's just that my experience with a language with great encoding support doesn't jive with what's claimed here.

Well I think that particular clause was written for C-style language users where the unicode support is definitely subpar.

Reply on news.ycombinator.com