HTTP Post Denial Of Service: More dangerous than initially thought | Hacker News Reader

HTTP Post Denial Of Service: More dangerous than initially thought

acunetix.com

42 pointsssclafani35 comments

This attack seems just as straightforward to address in a reverse proxy as "slowloris" (which was, despite the publicity, itself an attack known for many years prior to its announcement).

Two things remain true:

It is straightforwardly possible to bring a conventional full-featured web servers or app server to its knees with leveraged attacks (in which target pain greatly exceeds attacker effort).

and

This doesn't much matter, since even stupider attacks do just as much damage. For instance, during the 2002 Winter Olympics, which Arbor Networks was contracted to do anti-DDoS for, attackers simply got tens of thousands of machines to make vanilla browser HTTP requests to targets.

If anything, things like this "more dangerous than initially thought" request is easier to defend against than the dumbest attacks; it involves unusual patterns of POST requests, unusually large content-lengths, and unusually long-lived connections, all of which can easily make connections a candidate for a random early drop queue.

The fact that tools may not immediately exist to counter this kind of attack is probably not an indication of how hard those countermeasures are to build.

Couldn't agree more.

Anything that deviates from usual usage patterns can be relatively easily detected and negative impact mitigated.

But nowerdays botnets are the real threat with every 13 year old kid being able to get hundreds of machines infected with ease which is enough to pose a real threat to most websites when they launch absolutely valid HTTP requests. Botnet sizes of tens of thousands of zombies are not uncommon.

It is a tough problem and the reason why DDoS attacks are so effective and therefor popular. They work. When your site gets beyond a certain size, chances are you don't even have accesslogs for performance reasons because you get dozens of thousands of requests per second. So finding the malicious requests and singling them out from the rest to be able to filter them is hard to say the least. Plus every false positive could really hurt. Imagine you accidently filter some check-out requests in some online shop - that's a lost sale.

Attacks like the one outlined in the "paper" (it's really more a bunch of slides) can be easily defended against. They are not new just because someone alters the request method from GET to POST and wants to sell it as dangerous just to grab some fame. The fact that they suggest a change in the protocol would be needed just reinforces my suspicion. Announcing a flaw in a hugely popular protocol like HTTP gets you a sh*tload of attantion and in the end jobs.

I know a few people working for companies in the web-security field, some also member of OWASP. Exagerations are very very common. The more fear a customer has the more protections you can sell them.

The countermeasures you can use today to counter web server resource starvation attacks are:

  * Turn off keep-alives
  * Turn MaxRequestsPerChild/Thread down to a minimum
  * Set your timeouts low
  * Set your MaxClients/Threads to a number your server can handle
  * Enforce LimitRequest* settings
  * Enforce cpu runtime, memory, and other limits on CGI to prevent an overly-zealous set of web apps from loading your boxes
  * Don't accept POST and other methods if you don't need it, and enforce a maximum POST size in your web apps
  * Play with mod_antiloris, mod_qos and mod_evasive and see which works best for your application.
  * Use a single iptables rule to drop more than X connections at a time from any host

You're right that a stupid attack like a plain-old DDoS using normal machines to make normal requests is probably the most effective. After blocking tor and anonymous proxies there's only a few other ways to block a large number of different addresses from flooding your site, usually with packet analysis on # average connections per host and repeatedly requested URIs, but those aren't as trivial to implement as the settings above.

Any idea if "accept filters" will help mitigate this attack? See "man accf_http" on FreeBSD; not sure what the equivalents for other operating systems may be.

I know that varnish, Apache, and nginx support this feature. Speaking of... does having a reverse proxy like varnish help in these kinds of attacks?

Actually I wasn't aware of such a feature, it's kind of neat. But no, that won't help with this attack (the filter works only on HEAD and GET methods, and this attack is about the content after a POST).

It looks like this filter deals with reducing the CPU required to handle the request portion of an initial HTTP connection. I suppose this could help with preventing the server from being starved of resources for an attack on solely transmitting an HTTP request, but that could be mitigated by a 5 or 10 second timeout on getting a HEAD/GET request from the client initially (which should be reasonable even for dial-up and mobile connections). And I personally haven't ever seen a DoS attack on a proxy layer that ate up more CPU than any other resource, so I don't see this filter as being a significant advantage over the basic tuning you can do with connections on the proxies.

Oh, and as tptacek mentioned, a reverse proxy like varnish can definitely help in different DoS scenarios. I haven't used varnish in the real world so I can't speak to specific advantages, but in general, dedicated reverse proxies are a great way to offload initial processing, provide more resources for a high number of connections and deal with host-level attack mitigation so the app servers don't have to waste time on it.

Original paper: http://www.owasp.org/images/4/43/Layer_7_DDOS.pdf (well, slide deck)

§ Summary

➊ The slowloris attacks of last year consumed all of a host's available HTTP server connections with minimal bandwidth by trickling out HTTP headers forever at a slow pace. Countermeasures were developed and applied.

➋ This exploit consumes connections similarly, but does so by trickling out data in the content section of a POST request. Countermeasures targeting slowloris are ineffective.

➌ Countermeasures for this will include HTTP request timeouts, but to support distant users operating over potentially saturated links you will have to set the timeout high enough that any well connected attacker will still be able to kill your server. (That has some extrapolation from me.)

§ Discussion

The real solution will be to not consume a process or other expensive resource until the entire POST request has been received by the server. (Expand to PUT and any other method with content in the request.)

This becomes difficult in HTTP/1.1. To really support 1.1 you have to give the request the option to reject before the content is sent. (Yes you other old farts, the client doesn't necessarily blindly send the content, but waits for your server to say "ok, the method, resource, and headers look good, send me the content".) I suppose you could force all those decisions to be made by the HTTP server, but that would be sad.

To really support 1.1 you have to give the request the option to reject before the content is sent.

This is actually already specified, at least for PUT (see 100-continue).

This seems easy to defend against for sites that don't accept a lot of POST data. You could set a really low threshold for content-length. You could maybe even get fancy and only allow large POST data on certain URLs (e.g. low POST content-length threshold on www.example.com, but a higher threshold on admin.example.com; assuming that you can reject the connection as unauthenticated before receiving the data).

This makes perfect sense. If for example you're accepting 140 characters there is no reason you should have an excessive content-length for that URL.

This is not as dangerous as the authors make it sound like. Just limit the maximum connections per IP (e.g. via mod_evasive in Lighty or firewall like iptables) and voila - attack voided.

This is very similar to what Slowloris does but Slowloris is much more intelligent because it slows the connection down before it all request headers are sent and therefor many modules will not execute their limit checks just yet.

"What’s special about this denial of service attack is that it’s very hard to fix" "Therefore, to properly fix it would mean to break the protocol"

Not at all as outlined above.

From the paper: "In HTTP/1.1 where chunked encoding is supported and there is no “Content-Length” HTTP header, the lethality is amplified."

No, it changes exactly nothing regarding this attack. Why would it?

I do not know why these guys make such a fuss about something so trivial and obvious to anyone dealing with HTTP servers. But sadly this is quite common in the security industry.

All they need is a botnet of at least 256 computers, which can be rented for a small fee off any number of shady forums. Now what are you going to do?

A much better solution is to put a proxy in front that can hold lots of connections very cheaply. nginx and other evented servers can hold 10,000 trickling connections without using significant resources.

If you have a botnet attacking you, then it's a whole different story anyways. Those can get your services down easily regardless of what attack they use or what protocol is in play (HTTP,FTP,SMTP, you name it) so the botnet argument does not really hold here.

A proxy in front does not help if it does not limit per IP connections as an attacker can just use up all filedescriptors or connection slots. It just ups the ante a bit but not much. 10k connections can very easily be created from any DSL/cable connection let alone a server in a datacenter.

For most websites, sure. But the big boys have to deal with botnets all the time, and (mostly) manage to remain up. They need to figure this sort of thing out.

It still is not specific to this attack.

Their problem is still not specific to this attack. But they need to figure out how to handle every kind of attack that will come, including this one.

They do mention botnets deploying the attack, the "connections per IP" filtering will not help there.

I agree the chunked encoding statement makes no sense, nor does the "break the protocol" statement.

Some rough calculations (a simpler attack, with this timeout byte trickling isn't required):

Given: 200 bytes of header/transport can get a POST started and down to its content.

Given: A legitimate customer on a shabby, long latency link might take 5 seconds to get his POST request and data sent.

Compute: With a 5 second timeout, a bad person can hold a resource for 40 bytes/second of bandwidth.

Compute: 256[1] resource limit takes only 80kbps.

Compute: 20000 resource limit takes 6Mbps.

Still, you are right, the sky isn't falling:

• identify and throttle bad actors

• throttle single IPs (be careful about your giant corporate customer that NATs everyone)

• reduce the cost of the resource. Don't start a process until you have all the POST data. I would leave the data in the kernel until I had all the POST data or a lot of data (MSG_PEEK is your friend. 'real' requests will all come in the first datagram anyway) (Yes, kernel memory is special, but I sure don't want these attackers to cause swapping, so bytes are bytes and a billion of them are cheap.) Ultimately the TCP connection count will be the expensive resource. Google your way past the 64k limit.

• data tricklers are probably easy to see if you are watching the size of your read()s.

• crazy over the top solution: use a separate IP address on a virtual mac address in your linux box without a kernel IP stack on it. Do your own TCP blending the TCP state machine with your own HTTP request decoding state machine. (Actually, if you have slots, maybe just use another physical controller, it will be faster and you won't shoot your own leg off by accidentally engaging the kernel TCP stack. When you see a spurious RST in response to a SYN, look for that leg.)

EOM

[1] random self discovery of the day: I consider 256 to be one significant digit.

(This has been known about for ages.)

Yep, and this is why evented io is pretty much mandatory for a modern webserver (like nginx). If you need Apache you can always proxy nginx in front of it.

file descriptors aren't free, even for evented webservers. you've only moved the limit higher, not removed it.

This is true but not really responsive to the parent comment's point; the nginx architecture is better suited to handling this kind of attack; contexts are much more expensive than file descriptors.

File descriptors are themselves an artificial construct that can be coded around. In the unlikely event that this attack ever became super relevant, a relatively simple kernel module could drastically amp up the number of connections a host could absorb --- although a RED queue would probably keep you from having to go even that far.

I'm not sure what you mean by context. Clearly nginx also has some context, such as the data read so far, associated with each handle. I guess you mean thread stack?

Apache has an entire execution context associated with it (either a thread or a process).

I agree. I've been killing servers with it since the '90s. The current interest comes from it (possibly) arriving as an attack tool in botnets and not being mitigated by current anti-DDOS methods.

It is time to cope with it.

Yeah, because existing anti-DDoS methods are so effective at "conventional" DDoS attacks. Uh-huh.

And it would be easy enough to defend against if it really became a problem -- just time out. Most web sites don't need have a need to receive gigantic POST data.

It's difficult to distinguish between a persistent user trying to upload a large image over a spotty satellite link and a hacker.

You're right, but I still don't believe this creates an impossible situation:

* most webapps don't allow giant user uploads of any kind

* if you have a persistent user with a net connection that's that slow then you have no choice but to time him out

A sufficiently-slow connection will make even a reasonable amount of data take a long time. I'm sure Flickr has users that upload large photos over satellite links...

There's also no reason why you need to time this person out. If you configure kernel and application for small (or auto-sizing) buffers and use an event-based structure for the application, you should be just fine.

It's really a matter of one's approach and level of paranoia. Me? If I wasn't running Flickr (say I was running getfreckle.com), I'd probably opt for more conservative settings. I may not need to time that guy out, but I'd probably do it so I wouldn't have to worry later.

* If you configure kernel and application for small (or auto-sizing) buffers and use an event-based structure for the application*

This would be all the better.

True. Though it's a safe assumption that if you suddenly have a large number of spotty satellite users and they're DOSing your site it's probably something malicious.

Yep .. one example of a similar attack here - http://ha.ckers.org/slowloris/ - with references to others.

If this became a problem, couldn't we just have the server do triage? Like, monitor the speed of open connections, and when we're nearing saturation, tell some percentage of the absurdly slow connections (maybe with some heuristics to detect more likely attack patterns), "Sorry, we can't support you at this time." If we're going to deny service to someone, it may as well be the problem clients.

Use Apache's LimitRequest* directives http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestb...

This suggestion is impractical as the attack works with even fairly small Content-Length. Say 10000 and sending 1 byte per second will limit you to 10kb uploads but still give each connection hours to run. Since most IO timeouts are fairly large you can even with a limit of 1000 bytes run a connection for hours until you hit this limit.

The only solution is to limit connections per IP and maybe defining a minimum speed can help too.

acunetix.com

Reply on news.ycombinator.com