Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames?
1) Why?
2) Bye bye backward compatibility and interoperability
1) Why?
2) Bye bye backward compatibility and interoperability
It's already not really a thing.
Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f, and I think HFS+ requires valid UTF-16 and allows everything (including NUL).
The OS then adds its own limitations e.g. win32 forbids \, :, *, ", ?, <, >, | (as well as a few special names I think) and OSX forbids 0000 and 003a (":"), the latter of which gets converted to and from "/" (and similarly forbidden) by the POSIX compatibility layer.
The latter is really weird to see in action, if you have access to an OSX machine: open a terminal, try to create a file called "/" and it'll fail. Now create one called ":". Switch over to the Finder, and you'll see that that file is now called "/" (and creating a file called ":" fails).
Oh yeah and ZFS doesn't really care but can require that all paths be valid UTF8 (by setting the utf8only flag).
For just Windows -> Linux you can represent everything by mapping WTF-16 to WTF-8.
What I'm saying is that promoting UTF8 everywhere, without specifically stressing the fact that filesystems (in general) do no observe UFT8, leads to API/LIB designs that lack good support there.
Path/filename/dirname/whatever should be a different kind of "string".
Interoperability is quite possibly a good argument for coming up with some reasonable restrictions on filenames. Today you could easily (case sensitive names, special characters, etc.) create a ZIP file or similar that cannot be successfully extracted on this platform or that.
In an excellent article, David A. Wheeler [1] lays out a compelling case against the status quo. TL;DR: bad filenames are too hard to handle correctly. Programs, standards, and operating systems already assume there are no bad filenames. Your programs will fail in numerous ways when they encounter bad filenames. Some of these failures are security problems.
He concludes: "In sum: It’d be far better if filenames were more limited so that they would be safer and easier to use. This would eliminate a whole class of errors and vulnerabilities in programs that “look correct” but subtly fail when unusual filenames are created (possibly by attackers)." He goes on to consider many ideas towards getting to this goal.
[1] https://dwheeler.com/essays/fixing-unix-linux-filenames.html