That said, I am in the rowdy “Only UTF-8, ever, and nothing else” gang. Even thinking about UTF-16 gives me hives: life is too short. But Shift-JIS is OK.
wobble transform 8bit seems like an unnecessary hijacking of a well labeled error state
People had occasionally used the label before for mojibake of various kinds, but the term was never popular under that meaning. Simon’s work is now a vastly more popular meaning.
It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition.
Given that context, I don't see that UTF-8 actually helps much. Your fundamental structure has to look something like a rope of (byte sequence+annotation) entries. With that structure, using different encodings for different segments doesn't make things noticeably worse.
Stateful control characters make unicode mostly pointless - the whole point is to be self-synchronizing and have a universal representation for each character. (Granted emoji are busy destroying that already).
In Unicode 5.2, I think, this range was elevated from “strongly discouraged” to “deprecated”: https://www.unicode.org/versions/Unicode5.2.0/ch16.pdf#page=....
In-band signalling in Unicode in general is fraught. The Unicode 5.2.0 specification linked goes on to show various of the reasons why this sort of tagging is generally problematic and should not be used in normal text. (And this is why they were strongly discouraged from the start.)
Text direction signalling is another troublesome area of Unicode; there are multiple techniques, some strongly discouraged, and it’s a perennial source of interesting security bugs. The only reason direction signalling is supported at all is because it’s needed. Life would be easier with it gone.