Hacker News Reader

Comment by Iv | Hacker News Reader

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world.

I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540

and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and 0 O you still need to know a bit about control characters and you are good to go.

Unicode has tons of pages, control characters, diacritics and rendering oddities that make it hard to use like a tool.

I think your approach of considering like bytes to render is spot on. I consider these strings as something like SVG fragments: you can do text operations on them but you have to be confident that you know what you are doing.

View on HN

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII characters in programming that I can think of, but I think romanization can take care of it with most languages.

This, I think people are arguing for ASCII in programming, not going back to ASCII in general.

But aside from APL and a few Haskell coders, who isn't programming in ascii? I mean, sometimes i chuck an emoji into a comment but I don't think that really counts.

And how could programming in ascii have solved this bug? The problem is that the strings were compared without giving any thought to what "are these strings equal" is supposed to mean.

The only general solution to this kind of bug - that is, to considering distinct emails to be identical - is to have a special function that can check if two emails are identical according to the spec. This function will be different than determining if two names are the same in an American database where dotless and dotful i are the same, and that function will also be different than a function determining if two names are the same in a Turkish database where dotless and dotful i are different. Although I don't know why you're trying to compare strings for identity, you're probably just after similarity anyway - how often do we convert strings to lowercase before we compare them.

Generic string equality functions are always the source of bugs, since there's no useful, single uniformally applicable definition of string equality.

>I think romanization can take care of it with most languages

Except for the ones that it really can't. When you try and romanize standard written Chinese it becomes nigh impossible to read for most speakers, often it would be more confusing than an English translation.

I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but:

>there are quite easy ways to avoid homograph attacks

I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.

The simplest way to avoid most of them is to ban script mixing, especially mixing Latin, Greek, Cyrillic, and Cherokee.

So in a secure environment I can't write words in IPA? IPA requires mixing Greek beta and theta with Latin letters (but there's a Latin/IPA phi just for fun).

It also doesn't help ɡ vs g.

It sounds like any genuine solution is either not simple or excludes legitimate use cases.

Yes, the Unicode Technical standard identifies the Latin, Greek and Cyrillic (most of IPA characters) scripts as confusable, and recommends against mixing them in identifiers. Use in general text is fine though.

The Unicode Technical Standard [1] (different from Unicode Standard) recommends treating identifiers (filenames, variable or fuction names, email adresses, usernames, etc.) different from normal text. There is a special class of 'identifier characters' which already exludes a lot of sneaky characters like invisible punctuation and obscure scripts that are not in modern use.

Additionally there are 5 additional restriction levels for identifiers depending on your specific situation:

1. ASCII only

2. single script

3. single script or Latin+{Japn, Hanb, Kore}

4. single script or Latin+{any, excluding Cyrillic, Greek}

5. Not containing any characters in the recommended blacklist of characters for use in secure contexts

6. No restrictions other than normal identifier restrictions

For any specific strings that might be confused, there is an algorithm to compute the visual 'skeleton' of a string and match it against that of another string to test if they are confusable.

[1] https://www.unicode.org/reports/tr39/

> there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

No. There are these kind of attacks every now and then to this day. Maybe of you're not following itsec they fly under your radar, but getting this right is exceptionally hard. And besides these kind of attack, every major os had multiple bugs just in the processing of Unicode that could at least be used for DoS attacks.

So saying it's easy to avoid any sort of abuse of Unicode seems quite ridiculous.

Go ahead and support it for messages, display names and whatnot, but for the love of god, limit the login name of users to ASCII. Don't assume that your Python/Go/JavaScript lib for Unicode handles sanitizing and canonicalization properly. It doesn't. And even if it has only a minor bug that doesn't lead to direct issues, the next update of the lib might fix the problem and now you have to deal with the fact that your db might contain data that was processed with the old faulty lib and now gets compared to the properly processed output of the new version. Just don't. Use it as opaque data for displaying, as GP said, but never as an identifier for anything.

I swear software's biggest problem is that developers prioritize their own ergonomics over actually producing functional code. Imagine if auto engineers just decided "safety is hard, let's just make deathtraps". I know the current narrative is that the 737 Max failed because of MBAs but it's a really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash.

It was a bad corporate ecosystem that caused the crash. The people at fault were the accountants that tweaked lines un Excel until they got the numbers they wanted amd then pressed reality into the service of those numbers. Bad software didn't eliminate pilot training on the new systems. Bad software didn't shortcut the required FAA safety certification on the system. Bad software didn't reduce the number of redundant sensors that fed the system.

Bad beancounters did all that. C-levels in suits who got million-dollar bonuses for killing 346 people to gain marginally increased quarterly results. Don't blame the software.

> really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash

While I frequently point out how bad we are as an industry at making fault tolerant code, this part is just flat wrong.

The software portion of the 737, while definitely flawed in a catastrophic way, would not have came to be if the aeronautics engineers had done their job and designed a flight worthy plane without software hacks. Not to imply the aeronautics guys are the root cause either though, the 737 Max fiasco is a top to bottom completely failure of Boeing as a whole, virtually every department involved in the Max has a significant reason to share in the blame.

"I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages."

You are correct that most of the world is non-ascii and I am enthusiastic about recognizing that diversity and I am willing to pay certain costs in return for the richness it provides.

However, I will point out that certain systems have been deemed crucial and in need of deliberate (and brutal) dumbing-down. Specifically, I speak of the global Air Traffic Control system that is English only[1].

Tagalog/Flemish/Satsugu is hard. Let's (land airplanes with) English.

[1] https://en.wikipedia.org/wiki/Aviation_English

I want to be able to collaborate with Japanese, Chinese, Indians, Russian, Arabian, Malay programmers. Even if we somehow live in a dystopia where all the english-speaking world disappears in a magic puff, I think I will continue to shut down my native french and communicate with other fellow programmers in English ASCII.

I wish more programmers understood the value of having a lingua franca is.

UTF-8 should be used everywhere on the user-facing side, but under the hoods, you want unambiguous representation of strings and code. It is easy to avoid homograph attacks if you do have a non-unicode representation available to you.

The longer I program, the more I am convinced that falling back to ASCII in 90% of (non-embedded) use-cases is just a programmer avoiding to do the extra brainwork to deal with encodings.

That is why I like the Rust approach: make these issues front and center, implement clear solutions for common use cases and enforce them. In to many languages encodings feel like an afterthought rather than something that has been considered from the start. When I started using Rust I e.g. learned that the Strings a OS uses in its filenames are not necessarily valid UTF-8. Rust forces you to handle this explicitly. Strings are complex and hiding this can be dangerous

The issue I see with rust is that they (partially) repeated the mistake older operating systems made: they assumed that the Unicode of that time will be Unicode forever.

While UTF8 is just an encoding, a Rust UTF8 "String" is actually a UTF8-encoded Unicode 11.0 string. Or version 12.0, possibly 12.1. Maybe 13 soon! Who knows. A moving target, certainly.

If you're "agile", you can just recompile and you will be fine, the next Rust version will surely support any changes in the Unicode standard.

But once a Rust program or crate is no longer actively maintained, many small assumptions will be baked into its Unicode handling at a much lower level than, say, the typical Windows C++ program that uses the OS-provided dynamic libraries for string processing.

Who knows, maybe the consortium will never introduce breaking changes, but I suspect that in a decade or two people will be cursing Rusts too-strong integration with the Unicode of the 2010s...

The Unicode consortium has guaranteed that there are many properties they won't change once they're released. Even if the property value is objectively wrong.

More to the point: the standard library of Rust relies on no properties of Unicode that will change--there's no builtin normalization, case folding, grapheme cluster support, etc. So there's no data tables in the standard library that need to be updated. The only assumption Rust makes of Unicode is that no assigned codepoints will be changed, and that the space will not grow beyond the current limit of U+10FFFF.

I was thinking that. Unicode is great for presenting text to users but for file names, email addresses, code and the like ASCII has a lot going for it.

It's not just Unicode weirdness, it's the whole concept of string operations. Examples:

1) Naive CSV libraries that break when given a field containing a comma itself. Same for injections.

2) Indexing things by strings (file names, user names, etc) relies on humans typing and reading with 100% accuracy.

3) Control characters are still present. Try to generate a file name containing every ASCII character (from 0x01 to 0x7F), then try to delete it in different ways. It's really frustrating.

4) Even string templates can cause problems. MMO's used to have game masters identified with "[GM] Character name", until scammers started using the same pattern. The sin here was to concatenate "[GM] " with the character name, which is spoofable, instead of using a badge or different color.

For 3), that's not the case on Windows filesystems as these characters are not allowed in file and folder names.

On Linux, in addition to control characters, the shell interprets the `-` character specially. This means that if a filename starts with `-` you may have surprising results even if the filename is quoted.

> On Linux, in addition to control characters, the shell interprets the `-` character specially.

It's not the shell that interprets the leading `-` character specially, but rather the program receiving the string. The convention of marking the end of option processing with `--` helps, for programs that support it; you can also prefix relative paths with `./`.

No, it doesn’t. What’s a typical corporate email address? First.Last@...? F.Last@...?

More than half the world’s population does not use the Latin alphabet.

There's a certain value to say Arabs and Chinese being able to email customerservice@wherever.com rather than having to try to decipher خدمة الزبائن@ and 客戶服務@ in each others language.

Why not leave this choice for the providers?

Though essentially every language has a romanization.

Often multiple, which makes the suggestion rather less suitable.

Take a Japanese name like 麻生太郎. He may find his name romanized (ignoring the issue of surname/given name order) as Tarō Asō (preferable), Taro Aso (usually), Taroh Asoh (occurs on passports), Tarou Asou (‘I can't figure out diacritics’-style), Taroo Asoo, or even Tarô Asô.

Taro Aso would be the overwhelming choice for ASCII

And how's that supposed to work? Every country has developed some form of transliteration to the Latin alphabet. Its simple enough that anybody can memorize these additional 26 letters. A form of least common denominator. Anyone on this planet can scribble down their email address on a piece of paper and equally type one on their computer. Now imagine you'd write down a Chinese email address with characters. How am I supposed to type it? How's the Chinese going to type in the Arab's address? If I receive a mail from them, how can I confirm it's really them from the address? I can't read Chinese! But the Arab and the Chinese can read the Latin alphabet.

Case in point: we have punycode for over a decade now. China has their own native TLD. Still pretty much every Chinese website out there gets created under the cn TLD or even com, and uses Latin characters only. How come?

Counterpoint: A lot of people in China are as familiar with the latin alphabet as you are with the Cyrillic alphabet: Source: https://newrepublic.com/article/117608/chinese-number-websit...

I've once seen a Chinese girl texting on a bus stop. She used the ordinary US-looking on-screen keyboard but Chinese hieroglyphs were appearing in the message window as she typed.

Yeah my wife uses this input method too, as I do if writing Mandarin. It's basically typing the pinyin for the characters. However, nearly everyone else in her family uses the type of input where you type 'strokes' of the characters to filter down the character you want to input.

Sorry, this is wrong. They mainly use the US keyboard layout for input, typing in pinyin (mainland China. Taiwan etc. are indeed different). Every person using a computer not only has to know the Latin alphabet, but also has to know how the characters they want to input are translated into pinyin. Even on the smartphone where thanks to the touch screen you could easily come up with new flexible input methods, they simply ported that concept over.

This article doesn't appear well informed, or tries to make the Chinese look stupid. On a similar note there is the common belief that Chinese people, having simplified characters for well over 50 years now, couldn't read the traditional characters still in use in Taiwan and Hong Kong. They are still used for artistic reasons and in calligraphy, which a lot of Chinese people have practiced at some point in their lives. Based on my very limited sample size, I'd argue that a Taiwanese person has more issues reading the simplified characters than vice versa, but then again it's not like all the characters look totally different, so given at least minimal context it shouldn't be too hard going either direction. (end OT rant)

Yes, and good luck using a search engine to find 月山＠何でも.jp or الله@gmail.com

By the way, the first one is not a mail address as ＠ and @ are different characters. But actually it may be because ＠ is the @ in the Japanese character set. So it probably should be accepted right? How about if I use the Japanese ＠ in a korean address? Is that valid? Or a likely attack?

Also in the Arabic name, the first character of the string is the one closer to the @ (it reads right to left), be sure to take that into account in a search interface.

I think it is totally utopian to think that a programmer can know all the subtleties of all the writing systems in the world. I agree that this is a problem that needs to be solved, but users and UX designers should in no way underestimate the magnitudes of additional work it causes.

Reply on news.ycombinator.com