Hacker News Reader

Comment by kingkilr | Hacker News Reader

One point in and its already dead wrong, you never filter input, only output.

Edit: Everyone talking about databases: paramaterized queries, check them out.

Agreed, most input should not be filtered. There are some exceptions though like stripping whitespace and dashes from credit card numbers and phone numbers.

The filtering of input has nothing to do with security (that should be handled by output escaping and paramterized queries) and should only be used for improving the quality of the saved data.

I'd definitely filter input that goes into a database. But if you think the wording can be made better, definitely send a pull request. Thanks!

You've been downvoted but I completely agree. You cannot sanitize text for all possible situations; you can only escape it depending on the output format (database, html, etc).

These functions should be used to "validate" input rather than "filter" it.

I absolutely hate how for example phpBB creates garbled bbcode in the database because they thought that they should filter it or something, instead of doing it on output now every consumer of the data has to unfuck the content, because they messed up.

Input is almost always filtered. Without mincing the semantics of that, let's say you receive free-form NL query as input. You don't filter that to reduce it to core terms you send to the database? You don't remove prepositions? You don't tokenize at all? Input is ambiguous and reduced/filtered into pseudo-meaningful terms to return relevant output.

This is true for all structured data, but the problem always comes from freeform text fields, which you cannot filter for one context without mangling or destroying the data for all other purposes—except perhaps trivial things like trimming leading/trailing whitespace.

If I want to tweet

  I <3 O'Reilly books

you could pre-encode that as safe to paste into raw SQL, or as well-formed (X)HTML, but you can't do both simultaneously. Either encoding would end up distorting the content in the other context. You have to encode during output (and writing to a database counts) using the rules of the system consuming that output. Lots of crappy web forums visibly mangle punctuation in a futile effort to avoid this.

What a great example. I am going to remember that one.

What's wrong with filtering input? If you expect a number, what's wrong with telling the user "strawberry pie" is not a valid value?

There's a fine line between filtering and validating. What's meant with filtering is things like add_slashes and mysql_real_escape_string, a uniform transformation on the input, regardless of it's destination.

Validating on the other hand is checking if the values satisfy the expected requirements.

You only want to apply certain filters if you know they apply the current output. (htmlentities if output as html, mysql_real_escape_string if embedded in a query).

Filtering can't be done "regardless of destination", of course, since kind of filtering depends on the required result. Otherwise, difference between filtering and validation is largely in application semantics - some may prefer one, some another. Output is different things, of course filters can be applied to the output too.

What everyone is missing with the filtering argument is:

(1) normalization (things like trim, etc. so you don't do stupid things like invalidate an email address just because it was input with trailing spaces). The result is passed to a validator.

(2) validation (if a number is expected, it's only valid if given a sequence of digits within the correct range). Valid values are passed on to the next phase (likely persisted).

Sanitization is generally not needed and in most cases likely to cause data corruption. I'm sure there are use cases for it but it should be the exception, not the rule.

SQL injection attacks alone are almost always a result of not filtering input...

With parametrized queries, that becomes a non-issue, but I still see no point in cluttering the database with data that's just going to be filtered out at some point - might as well filter it before it goes into the DB to begin with. The exception, of course, being those rare cases when some users need to see the filtered data and others need to see the raw data, but even then, you likely won't want to allow everything.

I am not sure if you are referring to the same sort of filtering mentioned in the document. The data is filtered before it hits the database layer, this saves resource being used.

That's what I meant. I just kind of articulated it poorly, my apologies.

Parametized queries are just another way of filtering input ..

Yes, but on the site, they make a clear distinction between parametrized queries with PDO and other types of input filtering. I misunderstood and thought you were referring specifically to the latter.

No they're not. They're a method for ensuring that what is meant to go into some field in some database actually ends up there and does not start doing things it should not be doing.

It's sometimes known as sanitising input, so I'm guessing that's what they were trying to refer to.

In my anecdotal experience SQL injection attacks are almost as often caused by relying on filtering data and then forgetting to filter it for some fields. You should instead rely on parametrized queries and only use string operations to build queries where you must (and then do it very carefully).

SQL injections are very rare to non-existent in code where the programmers generally rely on parametrized queries.

This the wrong way to look at it, and yes, the PHP world always does this wrong because they're too focused on HTML.

SQL injection occurs when you're not escaping data while producing output, namely, an SQL query sent to the DB.

XSS attacks occur when you're not escaping data while producing HTML, but you don't need angle brackets to do it. <a title="$string"> allows for XSS injection with just a quote character.

Header injection attacks occur when you're not escaping data while producing HTTP/MIME headers, and all you need is a line-break character.

The escaping always depends on the output context, and the data in the database cannot be made safe for all these contexts. You will still need to allow people named "O'Brian", let people post "<_<" smileys, use Unicode in their names, etc.

> the PHP world always does this wrong because they're too focused on HTML.

No. The PHP world recommends, time and time again, using PDO and binding variables to queries. I've yet to meet an individual who does it the other way, other than people who are relying on extremely outdated tutorials (7+ years ago). Hell, even this document does. This document, unfortunately, uses the word filter in the wrong way, but the intention is still fine.

One thing I've run into is that PDO does not let you parameterize the field name in "ORDER BY" clauses. And there are a few other little things that I've wanted to do that it doesn't allow. It may be that me even running into these problems means I'm doing something wrong at a structural level. Regardless, in those statements where I want something to be dynamic, but it's not allowed by PDO, I do an exact equality check against a white list.

Well, that's not what binding is supposed to do. After all, field names aren't something you should be exposing at all.

> I do an exact equality check against a white list.

Pretty much the way you are supposed to do it (except, I'd actually use the value in the white list rather than the matched input value, as the two should not be the same).

I suggest removing the section about filtering entirely and making sure the section about PDO explains SQL injection briefly.

I'm not the op, but I'll keep that in mind fora push request. I think filtering just needs to be renamed to validation, and improved (but then that was the point of posting it here I imagine).

Which is fine for SQL injection, but completely ignores every other context, of which there are many, and which don't have PDO-like APIs. That's what I meant by PHP getting it wrong. The entire filter pipeline is built to assume you're outputting only to HTML, and it's useless for making data safe for other contexts without completely destroying it.

Output is not what goes to the user/page, in this context it is what your code sends anywhere, such as to mysql_execute or something

Reply on news.ycombinator.com