And how could programming in ascii have solved this bug? The problem is that the strings were compared without giving any thought to what "are these strings equal" is supposed to mean.
The only general solution to this kind of bug - that is, to considering distinct emails to be identical - is to have a special function that can check if two emails are identical according to the spec. This function will be different than determining if two names are the same in an American database where dotless and dotful i are the same, and that function will also be different than a function determining if two names are the same in a Turkish database where dotless and dotful i are different. Although I don't know why you're trying to compare strings for identity, you're probably just after similarity anyway - how often do we convert strings to lowercase before we compare them.
Generic string equality functions are always the source of bugs, since there's no useful, single uniformally applicable definition of string equality.
Except for the ones that it really can't. When you try and romanize standard written Chinese it becomes nigh impossible to read for most speakers, often it would be more confusing than an English translation.
No. There are these kind of attacks every now and then to this day. Maybe of you're not following itsec they fly under your radar, but getting this right is exceptionally hard. And besides these kind of attack, every major os had multiple bugs just in the processing of Unicode that could at least be used for DoS attacks.
So saying it's easy to avoid any sort of abuse of Unicode seems quite ridiculous.
Go ahead and support it for messages, display names and whatnot, but for the love of god, limit the login name of users to ASCII. Don't assume that your Python/Go/JavaScript lib for Unicode handles sanitizing and canonicalization properly. It doesn't. And even if it has only a minor bug that doesn't lead to direct issues, the next update of the lib might fix the problem and now you have to deal with the fact that your db might contain data that was processed with the old faulty lib and now gets compared to the properly processed output of the new version. Just don't. Use it as opaque data for displaying, as GP said, but never as an identifier for anything.
You are correct that most of the world is non-ascii and I am enthusiastic about recognizing that diversity and I am willing to pay certain costs in return for the richness it provides.
However, I will point out that certain systems have been deemed crucial and in need of deliberate (and brutal) dumbing-down. Specifically, I speak of the global Air Traffic Control system that is English only[1].
Tagalog/Flemish/Satsugu is hard. Let's (land airplanes with) English.
>there are quite easy ways to avoid homograph attacks
I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.
It also doesn't help ɡ vs g.
It sounds like any genuine solution is either not simple or excludes legitimate use cases.
Additionally there are 5 additional restriction levels for identifiers depending on your specific situation:
1. ASCII only
2. single script
3. single script or Latin+{Japn, Hanb, Kore}
4. single script or Latin+{any, excluding Cyrillic, Greek}
5. Not containing any characters in the recommended blacklist of characters for use in secure contexts
6. No restrictions other than normal identifier restrictions
For any specific strings that might be confused, there is an algorithm to compute the visual 'skeleton' of a string and match it against that of another string to test if they are confusable.
I wish more programmers understood the value of having a lingua franca is.
UTF-8 should be used everywhere on the user-facing side, but under the hoods, you want unambiguous representation of strings and code. It is easy to avoid homograph attacks if you do have a non-unicode representation available to you.
Bad beancounters did all that. C-levels in suits who got million-dollar bonuses for killing 346 people to gain marginally increased quarterly results. Don't blame the software.
While I frequently point out how bad we are as an industry at making fault tolerant code, this part is just flat wrong.
The software portion of the 737, while definitely flawed in a catastrophic way, would not have came to be if the aeronautics engineers had done their job and designed a flight worthy plane without software hacks. Not to imply the aeronautics guys are the root cause either though, the 737 Max fiasco is a top to bottom completely failure of Boeing as a whole, virtually every department involved in the Max has a significant reason to share in the blame.