Hacker News Reader

Comment by nodata | Hacker News Reader

Seems easy enough to guard against. Highlight the characters which are unexpected for my locale.

That's a clever idea, but I don't know how one would determine what "unexpected" is an increasingly international world.

nodata told you. Highlight characters not expected in my locale. My locale right now is en_US.utf-8. In security-sensitive contexts like file names and domain names, it's not actually that hard to figure out which characters are a surprise for me. And if I have a file that is named entirely in non-ASCII-subset characters... which I do... then light them all up. It's OK. It won't be hard for users to figure out what's going on, even if only subconsciously.

Code point has script property and mixing it inside strings which aren't likely to contain multiscript code points (like filenames) is a sign of trouble.

Filenames can contain multiscript code points.

E.g.: "Архив.gz" (Archive.gz)

But do default windows ones? I think (hope?) that the hosts file is hosts on every system, no matter which country or locale you selected - instead of, say, хозяин.

Perhaps limit system folders and files to ascii-only. Doesn't solve any of the picjpg.exe issues, but it's a start.

To be honest, I don't think it solves the problem of system file being modified by a process of unknown origin.

Then I could spell "hosts" entirely with cyrillic characters. (I'm not sure if it's possible with the word "hosts", but there's probably some word.)

Maybe you could. Anyway you could do it right now as well.

I doubt there is an acceptable non-heuristic solution.

how one would determine what "unexpected" is an increasingly international world

1. Highlight any two adjacent characters from different languages. Virtually no one needs an English word with one Cyrillic letter substituted in.

2. Highlight any word whose letters all look like they are for the current locale but is made entirely of code points from another language.

With a whitelist of multiple languages/ranges for those who can speak multiple languages.

But then what's the solution for someone who speaks both Russian and English?

Does Russian contain/mix English characters within words?

This is not a perfect solution because the RLO character isn't printed, there is nothing to highlight. There are several other non-printing characters in Unicode that I'm aware of, like the zero-width space.

I wonder if it would suffice to highlight characters not found in the primary font. I don't know how, exactly, fonts fallthrough, but I doubt most fonts venture outside locales.

Fonts increasingly venture outside of locales. Its very common for one font to have all western-European characters, or even all European (including Cyrillic) characters. This is especially true for the default system fonts.

CJK fonts often include not only CJK, but European characters as well. Almost always at least ASCII.

Ultimately, having a single font for all characters is desirable: Having to go track down more fonts because you're seeing � in your text is a pretty bad experience. Substituting other fonts is at best a kluge, as it often looks terrible.

Projects like DejaVu who plan to eventually cover all living scripts (http://dejavu-fonts.org/wiki/Plans) are not only a good thing but they are also making substantial progress.

Also, even in ASCII, there are a bunch of confusing characters (all depend on which font, of course): I (eye), l (ell), 1 (one), | (vertical bar); O (oh), and 0 (zero); {} (braces) and () (parentheses); 5 (five), S (ess), and $ (dollar), rn (r-n) and m; vv (v-v) and w; etc.

Wе nееԁ tօ fⅰnⅾ Ьеttеr ѕоⅼυtions tҺаɳ vіѕυаⅼⅼУ dіѕtіɳgυіѕҺing сҺаrаⅽtеrѕ.

Reply on news.ycombinator.com