Of course, I've also opined that Unicode supporting multiple encodings for the same glyph is also madness. The Unicode people veered off the tracks and sank into a swamp when they decided that semantic information should be encoded into Unicode characters.
As if that weren't enough, they also decided to cram half-assed formatting into it. You got bold letters, italics, various fancy-style letters, superscripts and subscripts for this and that.. all for the sake of leagacy compatibility. Unicode was legacy right from the beginning.
Unicode chose to support lossless conversion to and from other encodings it replaces (I presume it was important for adoption), so unfortunately it inherited the sum of everyone else's tech debt.
Note that nobody needs these code points to reverse text. I did it above without gnisu those code points.
[Purely-Latin borderline cases: umlaut (is not two dots in Fraktur) vs diaeresis (languages that use it are not written in Fraktur), acute (non-Polish, points past the letter) vs kreska (Polish, points at the letter). On the other hand, the mathematical “element of” sign was still occasionally typeset as an epsilon well into the 1960s.]
Unicode decides most of these based on the requirement to roundtrip legacy encodings (“have these been ever encoded differently in the same encoding?”), which seems reasonable, yet results in homograph problems and at the same time the Turkish case conversion botch. In any case, once (sane) legacy encodings run out but you still want to be consistent, what do you base the encoding decisions on but semantics? (On the other hand, once you start encoding semantic differences, where do you stop?..) You could do some sort of glyph-equivalence-class thing, but that would still give you no way to avoid unifying a and а—everyone who writes both writes them the same.
None of this touches on Unicode “canonical equivalence”, but your claim (“Unicode supporting multiple encodings for the same glyph is [...] madness”) covers more than just that if I understood it correctly. And while I am attacking it in a sense, it’s only because I genuinely don’t see how this part could have been done differently in a major way.
As a way to avoid going ASCII-only, though, excluding only things that look like syntax might be simultaneously not going far enough (how are homograph collisions between user-defined identifiers any better?) and too far (reliably transplating identifiers between languages that use different sets of punctuation seems like it’d be torturously difficult).
The idea of a letter in an alphabet and a printable glyph for that letter are two different ideas. Unicode could have and probably should have had a two-layer encoding where the letters are all different but an extra step resolves letters to glyphs. Where one glyph can represent more than one letter, a modifier can be attached to represent the parent alphabet so no semantic information is lost. Comparison for "same character" would be at the glyph level without modifiers, and we could have avoided a bunch of different Unicode equivalence testing libraries that have to be individually written, maintained, and debugged. Use in something like a spell checker, conversion to other character sets, or stylization like cursive could have used the glyph and source-language modifier both.
First off, Unicode cursive (bold, Fraktur, monospace, etc.) Latin letters are not meant to be styles, they are mathematical symbols. Of course, that doesn’t mean people aren’t going to use them for that[1], and I’m not convinced Unicode should have gotten into that particular can of worms, but I think you can consistently say that the difference between, for example, an italic X for the length of a vector and a bold X for the vector itself (as you could encounter in a mechanics text) is not (just) one of style. Similarly for the superscripts and modifier letters—a [ph] and a [pʰ] or a [kj] and a [kʲ] in an IPA transcription (for which the modifiers are intended) denote very different sounds (granted, ones that are unlikely to be used at the same time by a single speaker in a single language, but IPA is meant to be more general than that).
(Or wait, was this a reply to my point about Russian vs Bulgarian d? The Bulgarian one is not a cursive variant, it’s derived from a cursive one but is a perfectly normal upright letter in both serif and sans-serif, that looks exactly the same as a Latin “single-storey” g as in most sans-serif fonts but never a Latin “double-storey” g as in most serif fonts, and printed Bulgarian only uses that form—barring font problems—while printed Russian never does. I guess you could declare all of those to be variants of one another, even if it’s wrong etymologically, but even to a Cyrillic user who has never been to Bulgaria that would be quite baffling.)
As to your actual point, I don’t think the comparison you describe could be made language-independent enough that you wouldn’t still end up needing to use a language-specific collation equivalence at the same time (which seems to be your implication IIUC). E.g. a French speaker would usually want oe and œ to compare the same but different from o-diaeresis, but a German speaker might (or might not) want oe and o-umlaut to compare the same, while every font renders o-diaeresis and o-umlaut exactly the same. French speakers (but possibly not in every country?) will almost always drop diacritics over capital letters, and Russian speakers frequently turn ё (/jo/, /o/) into е (/je/, /e/) except in a small set of words where there’s a possibility of confusion (the surnames Chebyshev and Gorbachev, which end in -ёв /-of/, are well-known victims of this confusion). Å is a stylistic varisnt of aa in Norwegian, but a speaker of Finnish (which doesn’t use å) would probably be surprised if forced to treat them the same.
And that’s only in Europe—what about Arabic, where positional variants can make (what speakers think of) a single letter look very different. Even in Europe, should σ and ς be “the same glyph”? They certainly have the same phonetic value, and you always have to use one or the other...
Of course, we already have a (font-dependent) codepoint-to-glyph translation in the guise of OpenType shaping, but it’s not particularly useful for anything but display (and even there it’s non-ideal).
[1] https://utcc.utoronto.ca/~cks/space/blog/tech/PeopleAlwaysEx...
This is a total pedantitangent but I don't think that's actually true. These wikipedia pages don't talk about it directly but I think give a bit of the flavour/related info that suggest it's not nearly that set in stone:
https://bg.wikipedia.org/wiki/%D0%91%D1%8A%D0%BB%D0%B3%D0%B0...
https://bg.wikipedia.org/wiki/%D0%93%D1%80%D0%B0%D0%B6%D0%B4...
The second one, in particular, says early versions of Peter I's Civil Script had the g-looking small д, so these variants have been used concurrently for some time.
By the context!
How would a book distinguish `a` as in `apple` from `a` as in `a+b`? (Unicode has a separate letter a from a math a.)
By the context!
This is what I meant by Unicode has no business adding semantic content. Semantics come from context, not from glyph. After all, what if I decided to write:
(a) first bullet point
(b) second bullet point
Now what? Is that letter a or math symbol a? There's no end to semantic content. It's impossible to put this into Unicode in any kind of reasonable manner. Trying to do it leads one into a swamp of hopelessness.
BTW, the attached article is precisely about deliberately misusing identical glyphs in order to confuse the reader because the C compiler treats them differently. What better case for semantic content for glyphs being a hopelessly wrongheaded idea.
In fact, having a font that makes l I and 1 indistinguishable is plenty of good reason to NOT make this a requirement.