In fact this requires special language-specific knowledge anyway (which unicode provides in some tables and algorithms actually). In some languages ä should sort exactly as if it were 'a'. "aa", "äb", "ac". In others it should sort as a distinct letter (but not necessarily between 'a' and 'b'). Different Latin languages sort differently, I'm not sure if exact UTF-8 (or UTF 16 or UTF 32) byte ordering is actually appropriate collation in any latin-alphabet language.

But I do suspect it had something to do with ascii compatibility, I don't recall what. Very little of unicode is accidental, there's usually some reason for whatever in it.