Comment by jerf | Hacker News Reader

Comment by jerf | Hacker News Reader

Parent Full thread

I can't emphasize this enough. This isn't a matter of taste, like, maybe you sanitize, maybe you escape on the way out, it's all good, it all works, it's just a matter of opinion.

Sanitizing the input is wrong. Actively, objectively, unrecoverably wrong. Once you've destroyed your data you can't get it back. Huge amounts of effort have been wasted by people trying to fix and recover data that was destroyed by systems "helpfully" "sanitizing" data. God help you if you have a sequence of these systems in a row each doing their own "sanitization" before you get the data.

Do not "sanitize" your inputs. Do not tell other developers to sanitize their inputs. Do not sagely spout off on HN about the importance of sanitizing your inputs. It is wrong.

The only "sanitization" that should be done is that when encoding to the output there are sometimes things that should simply be removed. For instance, a good HTML escaping function probably ought to entirely drop nulls, not even encoding them as  or anything, just drop them. Some of the other ASCII characters are straight-up illegal in HTML as well, even encoded. But all that sort of "sanitization" should be in the escaping step. If you want to reject null characters at input time, that's part of validation, not sanitization.

_Some_ sanitisation is fine. For instance, stripping leading and trailing space in some fields, case normalisation, automatic insertion of spaces in credit card numbers, that kind of thing. That is to say, you should sanitise as an affordance to the user. Given the choice between presenting an error to the user and automatic sanitisation, the latter is preferable. It's something that should be done carefully, but it's still good.

Thoughtless sanitisation is a whole different kettle.

I agree that cleanup is acceptable, and there's certainly some wiggle room in what people call cleanup vs. sanitization and such.

But when people chant "sanitize your inputs" and expect it to be treated as sage wisdom, it's in a security context, and it is wrong in that context. Sanitization is not a valid security tool. Mind you, you might be forced into it if your back is against the wall and you're working on other code that is broken and you can't fix that other code's broken failure to escape or whatever. But it's still wrong, just a wrong thing you were forced to do.

A richer point of view is more "don't destroy data you don't 100% mean to destroy". Whitespace in the wrong place or stray nulls can meet that bar. Removing characters for "security" reasons doesn't. Destroying data to prevent security issues downstream is not a good idea.

To me that sounds more like canonicalization than sanitation. Depending on your requirements it might be fine to convert the input to a canonical form before processing. If you do this, be certain to do it before validation so that you don't accidentally "canonicalize" validated input into something which wouldn't pass the validation checks.

A key aspect of canonicalization compared to sanitation is that the result should be something that the user would consider equivalent to their original input. The most common offender in my experience is the abuse of case normalization, especially for data like email addresses which are not defined as case-insensitive (at least for the mailbox name) even if many servers treat them that way. If you don't preserve the original case (and other parts such as "+" labels whose meaning is defined by the mail server) the address may not work at all, or may result in sending messages to the wrong user.

Names, as an intimate part of the user's identity, are another area where case normalization can sometimes prove annoying or even offensive. If some legacy system requires names to be entered as all-caps US-ASCII characters, fine, but at least don't turn "O'Conner" or "MacDouglas" into "O'conner" or "Macdouglas" in some misguided attempt to ensure that just the first letter is capitalized. (And in some situations the first letter shouldn't be capitalized, e.g. the "dos Santos" in "Giovani dos Santos Ramírez"[0]—which is a single surname, not two names.)

[0] https://en.wikipedia.org/wiki/Giovani_dos_Santos

Oh, believe me. As somebody with a name that includes accents, and a surname that contains two words, with relatives whose names include internal capitalisation and apostrophes, I know _all_ about that.

The thing is that canonicalisation is a kind of sanitisation. As you mentioned, I personally prefer it to be done in real time. Sometimes it can't, however, you have to resort to munging, which is on the nastier end of sanitisation. Here's a short story:

AFNIC run the .fr registry, and they, unlike other registries, expect you to provide a contact's given name and surname separately. The joys of French bureaucracy. At my previous job (hosting provider and domain registrar), I built the company's domain management system. The systems in front of that didn't care about the form a person's name took so long as it was present, and most other domain registries are the same. There was no sensible way to get the applicant to enter them previously (this data was taken from the billing system). This necessitated that I build a library that could parse people's names, and I ended up developing a rather large number of heuristics for doing so as accurately as possible. It only covered the Latin alphabet, as that's all AFNIC would accept at the time, but it worked.

The problem is that most don't put that kind of thought into data sanitisation, and do things such as those you mentioned. And that's why we can't have nice things.

If I'm reviewing code and someone is implementing escaping that's an immediate, massive, red flag. It's SO HARD to get right and there are many MANY libraries for doing it correctly. The scary thing is how many bugs still make it into these libraries.

Strongly prefer using an established library and see designs such as https://web.dev/trusted-types.

The downside of not sanitizing inputs is that the data may well end up in contexts or new apps altogether wherein escaping output is not reliably done or known to be necessary. In an ideal world it shouldn't happen. But, it does, especially given turnover within organizations and the long lifespans of many systems and their datasets, lack of documentation, etc.

So, this calls into question the viability of a strict policy for every scenario that encourages blithely storing <script>doEvil()</script> in the database.

Seems some level of sanitization has a place as an extra layer of defense for some use cases.

> Sanitizing the input is wrong. Actively, objectively, unrecoverably wrong.

I agree on the “unrecoverably” (sic) part, but strongly disagree on words like “objectively”. It can be bad only if the input sanitization is poorly done. If that’s poorly done, then it’s also likely that the output sanitization may be poorly done. One cannot then say that output sanitization is objectively bad because someone doesn’t know or care enough to do it properly.

This is a complex topic that deserves more attention, not hand waving away with claims that cannot stand on their own.

Validate inputs, escape outputs

Yes, but remember in a lot of cases nearly anything is valid input.

Reply on news.ycombinator.com