Hacker News Reader

Comment by tomxor | Hacker News Reader

I agree with the overall sentiment of your comment but..

> It's exclusionary and for zero good technical reason. For every reasonable programming language, there are functions to do everything in UTF-8 that you can do in ASCII, with about the same complexity.

At minimum, unicode does come with a lot of complexity in implementation, and a lot of potential vulnerabilities due to that complexity.

Not saying that excuses all websites, modern DBs handle unicode just fine after all, and the browser takes care of the rendering - (although some might crash your computer), and there are usually builtins for serverside web languages to help you sanitise strings... still, things would be way simpler with any fixed word <8bit encoding without glyph manipulation builtin. So yeah, it's complex, but it's usually worth it.

View on HN

Unicode includes a lot of features that aren't universally desirable in applications that handle text. "Zalgo" text (many stacked combining subscript and superscript characters) can break out of the box it's contained in. Emoji characters render as distracting full-color icons that are easily confused for UI. Directional overrides let you plop an "everything after this should be printed backwards" control character into any string you control, providing endless opportunities to break UI and confuse other users, as in https://blog.malwarebytes.com/cybercrime/2014/01/the-rtlo-me...

IMO websites like this that challenge the assumption that all modern software is obligated to support all of Unicode are a valuable contribution to the world, if only because they might inspire the creation of a better, more well-scoped character set.

> IMO websites like this that challenge the assumption that all modern software is obligated to support all of Unicode are a valuable contribution to the world, if only because they might inspire the creation of a better, more well-scoped character set.

Oh God no.

We do not need any more character sets.

Most of what you mention can be fixed with a font, and the LTR-override stuff is in the realm of proper escaping which all web apps everywhere in the world need to do at all times absolutely goddamned regardless.

Thank you

I was in a camp with many other comments here, that this is 21st century and supporting UTF-8 should be a default.

But that malware example makes me almost do a 180. Who in their right mind thought that full Unicode should be allowed in file names?

Is there a middle ground? Text-only bits of Unicode?

It's genuinely pretty tricky. There seems to be a valid use case in filenames if it needs to include both an English and Arabic or Hebrew component, for example. Which, if you work with bilingual documents between countries, is not unusual.

It's part of a broader question of visual strings having multiple unexpected Unicode representations, including Cyrillic letters masquerading as Latin, etc.

I've never heard of any solution, except for when a text component is supposed to be machine-interpretable (e.g. a file extension or domain), for the computer to display its interpretation in a special way (via icon, showing the extracted extension below the name, bolding the domain, etc.).

They're not great solutions, but there don't seem to really be any alternatives either.

> unicode does come with a lot of complexity in implementation

As far as I know, ASCII is a proper subset of UTF-8. That makes implementation quite natural in most cases (e.g. string replacement).

EDIT:

Looks like Lucian built this site in PHP, which easily handles UTF-8, as well as most DBs.

I don’t know if you use Android based phones. It is common for these devices to practically never get updates. I think there are a lot of users stuck on older versions of Unicode?

Reply on news.ycombinator.com