Hacker News Reader

Comment by Someone1234 | Hacker News Reader

Can someone explain to me why case insensitivity is a bad thing? Clearly Linus believes so but didn't explain why he believes so.

Most UNIX and Linux systems seem to have an "all lowercase" or "all uppercase" convention, so the fact that they have case sensitivity is often not utilised.

In fact the biggest reason you'd want case sensitivity off the top of my head is legacy support but that's just a circular argument (since you never really reach WHY it was that way originally, just that it was).

I guess based on what he talks about next he is worried about how case insensitivity interacts with other character sets (i.e. does it correctly change their case), but for most sets isn't the lower and upper case defined in the UNICODE language spec itself?

View on HN

The prime number-one concern in kernel programming is managing complexity. Well, in most programming really, but in kernel programming unmanaged complexity leads to lost data and sometimes broken hardware instead of "just" crashes.

Case-sensitivity is the easiest thing - you take a bytestring from userspace, you search for it exactly in the filesystem. Difficult to get wrong.

Case-insensitivity for ASCII is slightly more complex - thanks to the clever people who designed ASCII, you can convert lower-case to upper-case by clearing a single bit. You don't want to always clear that bit, or else you'd get weirdness like "`" being the lowercase form of "@", so there's a couple of corner-cases to check.

Case-sensitivity for Unicode is a giant mud-ball by comparison. There's no simple bit flip to apply, just a 66KB table of mappings[1] you have to hard-code. And that's not all! Changing the case of a Unicode string can change its length (ß -> SS), sometimes lower -> upper -> lower is not a round-trip conversion (ß -> SS -> ss), and some case-folding rules depend on locale (In Turkish, uppercase LATIN SMALL LETTER I is LATIN CAPITAL LETTER I WITH DOT ABOVE, not LATIN CAPITAL LETTER I like it is in ASCII). Oh, and since Unicode requires that LATIN SMALL LETTER E + COMBINING ACUTE ACCENT should be treated the same way as LATIN SMALL LETTER E WITH ACUTE, you also need to bring in the Unicode normalisation tables too. And keep them up-to-date with each new release of Unicode.

So the last thing a kernel developer wants is Unicode support in a filesystem.

[1]: http://www.unicode.org/Public/UNIDATA/CaseFolding.txt

That's a really good and comprehensive answer, thanks.

I second this. I was trying to understand the reasons and your answer helped me. It saved me a few hours. Thank you!

Well put.

However, I'd add that what sucks even more is that this is not just a kernel issue. Every single application needs to worry about this.

In this particular instance, the vulnerability is caused by git not handling aliased names correctly.

And even if you do things The Unicode Way, you'll get a constant stream of bug reports from people who don't like The Unicode Way, and want things done Their Way, which is, of course, The Right Way.

There are as many Right Ways as there are people, of course, and no two are completely compatible, so if you want to Do The Right Thing, either go see a movie or write your application software to do it like that, because the OS kernel can only do it one way. And, of course, if you want applications to be able to do case folding, the filesystem needs to be case-sensitive so all the relevant information is preserved.

"And keep them up-to-date with each new release of Unicode"

I agree with the rest, but that is one thing you absolutely shouldn't do. Once there are disks 'out there' that were created with some idea about what case insensitivity is, your choice has been set in stone. The risk of dhanging any rule is simply too high. Somebody might start reading hat disk using the previous version of the file system.

For example, the precursor to HFS+, HFS, like HFS+, kept directories sorted by name. However, it had a filename sorting bug that the Finder had to work around (see http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.ht...)

> I agree with the rest, but that is one thing you absolutely shouldn't do.

And now you have a file system that is case sensitive, BUT ONLY IN SOME UNICODE RANGES. Somehow, that's better than just not being case sensitive in the first place?

Mac OS X's NFD conversion was the problem for me. The case insensitivity was, superficially anyway, not the problem - you get back the filename you initially wrote (it just could overlap with another file that is not clearly defined because world languages are crazy). But NFD changes the filenames you wrote into a different byte sequence.

I've use all linux systems for a while, recently added a macbook pro retina. I keep some personal directories of stuff backed up on multiple systems, along with a "manifest" of files and md5sums.

NFD conversion meant that file names, which included correctly-encoded utf-8 (NFC ...) accented characters, changed between writing them, and "ls" of the directory.

If I didn't already have experience with utf-16 and all sorts of unicode encoding and translation silliness, debugging this would have been impossible. The fix was to add this to my scripts:

| perl -M'Unicode::Normalize' -M'open qw(:std :utf8)' -ne 'print NFC($_);' |

man oh man was that hard to figure out though. Mac OS X is way way harder to make work right than linux. (See also disk utility)

I get the problem, but don't you need canonical equivalence to make sure you don't allow files with filenames that look the same but which happen to have a different unicode byte sequence?

Or is the problem specifically with NFC? I don't see how byte equality could be preserved the other normalisations.

With or without Unicode's own meddling, characters may still look the same. Think of non-Latin alphabets that have letters defined in an identical fashion to those in Latin alphabet. Heck, even without other alphabets, the Latin letter O and the digit 0 are valid characters for filenames in any system I know of, and although looking the same they are yet different. (Having a bar, a point, or some other distinctive mark inside 0 is just a hack, the common description for the aspect of this digit makes no distinction from letter O.) This problem is not new, definitely not brought up by this Unicode thing.

True, we have seen this issue with fraudulent domain names, where they can pose a security issue.

To me it seems normalisation at least reduces the attack surface, while not solving the issue completely. I am not qualified to judge whether or not the complexity increase is worth it. But what are the downsides - what is so bad about NFC?

Well in my case, I don't have to worry about filename spoofing, or even filenames being too similar, for my own directory of records/archives.

The strangeness of a normalizing filesystem, is that you can try to open a filename (with accent, NFC), and it works. Then, a readdir() doesn't list that exact filename byte sequence (you happen to get one that's extremely similar, but NFD).

On linux it always does, there's no normalization, exactly what you open is exactly what's listed. This really is an awefully nice simplification, for this layer of abstraction. But you could argue that I'm just accustomed to the tradeoffs of it by this point.

Speaking from imperfect knowledge, I'd guess: case insensitivity implies allowing for aliasing (case aliasing for case insensitive and god-knows-what for Unicode insensitivity.)

Which means that anywhere you handles names, you explicitly handle these aliases. Miss a spot (or an alias), and you have a security bug.

In this case, the specific exploit Linus is referring to works by commiting a malicious .git file as .Git to your repo. Then somebody else on OSX clones your branch, and the bad .Git file will overwrite your .git file, causing a security breach.

Yes, it would be nice to have universal "uppercase" and "lowercase" rules, but in the real world, collation rules are crazy, arbitrary, and damn near impossible to get right. Now try to get it right in EVERY locale around the world, because if you don't, you open a big gaping security hole or data corruption bug.

Indeed "uppercase/lowercase" is a pretty anglocentric way phrase things. Who know what sort of cases somebody might come up with in their next language. "Normalization" is properly a more general term, but who says a normalization always exists?

I don't know to what extent this affects HFS+, but case insensitivity in general is locale dependent. The classic example is the Turkish distinction between dotted and dotless I - http://en.m.wikipedia.org/wiki/Dotted_and_dotless_I.

Quote:

The dotless I, I ı, denotes the close back unrounded vowel sound (/ɯ/). Neither the upper nor the lower case version has a dot.

The dotted I, İ i, denotes the close front unrounded vowel sound (/i/). Both the upper and lower case versions have a dot.

End quote.

I can imagine that being a nightmare, particularly if one user uses a Turkish locale and another on the same machine uses English. And all of that complexity in the kernel? Ouch. (Or HFS+ just behaves incorrectly for Turkish file names, not sure that's really better?)

Upper/lower case is locale language sensitive. One specific example, the uppercase for "i" varies depending on whether you have a Turkish locale or not. String.toLowerCase() in Java has exceptions coded in for the "tr", "az", and "lt" locale languages. Some of these languages may cause the length of the string to change when upcased or downcased, too.

Additionally, given a utf8 string, OSX will translate it into another, possibly different utf8 string before using it as a filename (the NFD normalization that Linus mentioned).

The exploit being referenced allows malicious code to be injected into a file that handles git, which is (.git). Now, normally Git does not allow you to overwrite (.git). Since HFS+ is case-insensitive, writing the file as (.Git) will overwrite the (.git) file and break security. Linus is saying that an uppercase and lowercase character are two different things, and not recognizing that can (and did) cause problems with security.

His argument is wrong. The issue is not with the filesystem being case-insensitive; the issue is that some strings which are not identical map to the same filename.

Two strings differing in case is only one such example. Another example is the presence of codepoints which are ignored by the filesystem. The fix for git and Mercurial included the fix for this as well. Does Linus think all filesystems should allow all 256 byte sequences in every permutation as well? (He might, since that is the behavior of common Linux filesystems.)

The proper mindset is to realize the actual issue (multiple strings map to the same filename) and use a "filename_according_to_program -> filename_according_to_filesystem" function everywhere when dealing with filenames, not to blame the filesystem.

You hit the nail on the head, he does expect things to work almost exactly as they do on Linux. The thing about Linus if you've ever spent any time on reading the Linux Kernel mailing list is that he is extremely explosive and at times has really harsh analogies. But...he is usually right.

The question you need the filesystem to answer in this case is, "If I save a file with name A, will it be accessible via name B?" You can't assume that the filesystem has one canonical name from which the file will be accessed.

EDIT: That actually isn't even enough, because the answer may vary by locale.

Even worse, to solve this for Git's case, you need to know the answer

1) For every locale

2) For every supported filesystem

I've run into issues with it from a programming point of view, you have to make sure to do a (slightly) more expensive case insensitive comparison when dealing with filenames.

This is only an issue when it is fake case insensitive. In the sense that it claims it is case insensitive but in fact is case sensitive (and returns case sensitive filenames). NTFS is a perfect example of this, it claims to be case insensitive but in fact stores everything in a case sensitive way (so you wind up doing things like ToUpper() on every filename).

On a legitimately case insensitive filesystem both "etc" and "ETC" would return identical filenames as they would be stored identically in the file system's database (e.g. would return "etc" for both).

So what you're really giving is not an example of why case insensitivity is bad, but why half measures are bad.

>NTFS is a perfect example of this, it claims to be case insensitive

Where does it claim this? In any case this is not true because...

>but in fact stores everything in a case sensitive way

This does not have to be true. NTFS can be case-sensitive or case-insensitive depending on which subsystem is accessing it. For example, while the POSIX subsystem for Windows was a thing, installing it had the side-effect of turning NTFS case-sensitive.

Pro-tip; never do .ToUpper() / .ToLower(), use foo.Equals(bar, StringComparison.OrdinalIgnoreCase), saves a string allocation also only loops over the strings once. For the truly anal.

Reply on news.ycombinator.com