Google web security expert posits the "Millenium Prizes" in web scanning
lcamtuf.blogspot.com
lcamtuf.blogspot.com
There are wildly different opinions about how useful/effective these tools are, despite the fact that they generate tens of millions of dollars of revenue and arm a significant percentage of all web app testers (we've banned them at Matasano, for whatever that's worth).
HN usually doesn't care about inside-baseball appsec stories, but I think there's an opportunity for HN to geek out on the big unsolved problems Zalewski --- one of Google's leading security people --- suggests afflicts the space:
* How do you compare HTTP/HTML responses to detect templated versions of the same basic page structure (pages that differ only in the database row used to generate them, but with the same inputs and the same mapping of inputs to outputs)?
* How do you even detect whether a page is valid or not? Most large web apps intercept and articulate the page-not-found exception, so that it looks like any other part of the app.
* How do you figure out how to reliably and automatically feed inputs to 100 different applications that will encode inputs totally differently?
* How can you distinguish a "500 Internal Server Error" caused by an uncaught exception handler in defensive code from a "500 Internal Server Error" caused by an undefended component crashing "exploitably"?
There's pretty good money in a simple, clever, effective answer to any subset of these questions, for what it's worth.
How would you solve it as a screen-scraper?
I've used that trick restoring a database when a sysadmin on some site went rogue.
Very effective.
I think you could probably get some value out of a side channel attack like timing (this is essentially isomorphic to "I can't log you in [because your password was wrong on the first character]" versus "I can't log you in [because your password was wrong on the second character]", except with several orders of magnitude more difference in response times between two textually equivalent responses).
For example, it should be absolutely trivial to determine whether a given input made it past the web application (to the database, underlying web service, remote API, etc) or not. Whether that results in useful information, who can say.
I'll occasionally use it for what I think is a straight-forward purpose, and more often than not, it's more trouble than it's worth. We've demoed a couple of other web scanners, and it definitely doesn't seem like WI is alone in it's awfulness.
His broader point is that you can't count on ?a=b&c=d formatting if you're writing a tool that needs to exercise any web application it finds. There are definitely some goofy alternatives to that format lurking out there.
This, naturally, broke the existing query string parsers, but never fear, they built a custom string tokenizer (which did not actually descend from StringTokenizer) which segmented the string around "amp;"
It was brought to my attention after a lecturer in web design at $CLIENT couldn't post a homework assignment for his students because it included "&" in the message body text field.