If unicode tr didn't exist, then by fixing tr you fix this problem for everybody, not just yourself. You're only in trouble if the source for `tr` isn't available. But the source for `tr` has always been available, even for proprietary unices.
As I understand it, that was the original spec, which both Knuth and McIlroy wrote to. I agree that it is limited as you say.
> The shell solution, built upon standard tools, can not be extended to work in an international context but a custom solution in Python (or even Pascal) quite concievably could.
As bryanlarsen pointed out, the shell solution can easily be extended by using an internationalized version of tr. The Python equivalent would be to use the built-in Unicode support. (If Pascal had that, you could do the same in Pascal.)
However, it's worth noting that by specifying the problem that way you still have the issue of how the input stream (which is going to be bytes) is encoded. Essentially, the original spec declared by fiat that the encoding was ASCII.
Also, btw, you can express non-English languages in ASCII (though certainly not as wide a variety as in Unicode); the program as written does assume that words are composed only of the 26 standard ASCII letters, but it could easily be extended to include the ASCII special characters. Another exercise for the reader. :-) Though if you're going to do this kind of extension, it might be better just to go the whole way and handle Unicode.
> As bryanlarsen pointed out, the shell solution can easily be extended by using an internationalized version of tr.
You should try that and then blog about it. :) I've spent lots of time battling issues with ascii-centric libraries. My conclusion is that it is not easy at all which is not strange because tools like tr and sed were written decades before unicode support became a must have. I can't say that it is impossible to write a shell script to count words in a text written in Arabic script, but it doesn't seem easy.
Where did I assert that?
> I can't say that it is impossible to write a shell script to count words in a text written in Arabic script, but it doesn't seem easy.
Is there a definition of what counts as a "letter" in Arabic script? That is, which Unicode code points correspond to letters? And does Arabic share the convention that a "word" is a sequence of letters delimited by non-letters?
If the answers to those questions are "yes", then the extension of the algorithm already presented to handle Arabic is straightforward; it's just a matter of substituting the Arabic definition of "letters" for the ASCII definition. (With, as I said in my previous comment, the additional issue of determining the encoding of the input and output.)
If some of the answers to the above questions are "no", then you have an issue with the problem specification, not with the algorithm you're going to use to solve it.
My new opinion is that great programmers will go beyond the spec and make their software handle Unicode in cases like this. It's the right thing to do. If the spec didn't require units tests does that mean you don't write unit tests? No, you do it anyway because it's the right way to do things. BTW I'm still working on being a great programmer myself.