SYN Flood Mitigation with synsanity | Hacker News Reader

SYN Flood Mitigation with synsanity

githubengineering.com

67 pointsalanfranzoni17 comments

tl;dr: They backported an existing kernel feature (per-cpu syn cookie generation) to 3.x using iptables (rather than just maintaining a patchset) because they didn't want to run a current kernel or maintain a fork.

I am curious how effective this is at larger synfloods. Obviously when something hits near 1gbps, its game over for any server as that floods the uplink (assuming a standard 1gbps hookup)

How many mb/s or pps can this handle on a average server while still having the server be able to respond to legitimate requests? ( lets say a average server is a quad core, 3.5ghz , 8gb ram)

I believe nowadays DDoS defense has to be in depth:

- Anycast to divide the attack among different datacenters

- Multiple filtering devices with +10Gbps uplinks at the edge

- Then things like SYNPROXY and this synsanity at the final servers

My point being: I don't think asking how effective is synsanity against larger synfloods is a complete question.

Speaking as someone who ran a very popular video chat service that had on average 5+ DDoS attacks a day, I can tell you that the vast majority of our attacks were under 1gbps in size. Usually the 'attackers' would use services you can rent on hackforums for 5-10$/week to 'boot' websites and people offline.

Thats why I was asking about how efficient this was on a per-server basis and why I think a answer would be beneficial. Most DDoS attacks are not the multi-gig reflection attacks you read about and defending against those types of attacks is much more complicated.

Regardless, I am not debating if ddos prevention has to be in depth or not. Im asking how efficient this is on a single server.

Gotcha! thanks for your insights :)

Out of curiosity, why bother with SYNPROXY if your edge filtering device can weed out the SYN floods? Legit question, no trollo.

Also, anycasting to deflect an attack is easier said that done. It works, but it takes a lot of work to get there, and you really have to be well coordinated and planned out.

Since SYN packets only contain a limited set of information, if the SYN packets have spoofed source addresses then it is very difficult for a device in the destination network to filter/mitigate a SYN flood, since they look like legitimate SYN packets from many different clients. That said, if it's a non-spoofed attack, then you can definitely filter them at the edge. For spoofed attacks, if you filter from multiple global POPs (as do many DDoS scrubbing services), you may be able to guess that a packet is arriving at an inappropriate POP given the source address, but even that will only let you filter a certain amount of the traffic. Because of that, you still need some level of protection at the destination server.

Agreed. For the large SYN floods (X Mpps hitting a single machine) syncookie solutions don't work. What works is identifying the SYN flood itself and filtering it.

Generating SYN cookies in Linux doesn't work well at those rates without help. That said, it's very possible. Solarflare cards are good at withstanding several million PPS SYN floods...provided you license and then extend their IP stack with the card.

Many hardware load balancers can do +100M SYN cookies per second (assuming a proxy/gateway deployment instead of DSR).

So SYN cookie solutions do work; they just don't work great on stock Linux.

Edit: Add clarification about LB deployment. Won't work with DSR.

That said, it's very possible. Solarflare cards are good at withstanding several million PPS SYN floods...provided you license and then extend their IP stack with the card.

Agreed. We use Solarflare cards but we don't syncookie when we get a large flood. We identify the flood and drop it. There's little point firing back millions of useless SYN ACKs per second.

Agreed. We use Solarflare cards but we don't syncookie when we get a large flood. We identify the flood and drop it. There's little point firing back millions of useless SYN ACKs per second.

How do you know what to drop without SYN cookies? If the attacker isn't advanced, sure all the SYN's will look the same. What if the attacker is smart and the SYN's look legit and are spoofed? If you drop those SYN's, you'll run the risk of also dropping legit traffic if you don't use cookies.

I don't know very much about defending against DDOS attacks. How do you identify a SYN flood and filter it without interfering with legitimate traffic?

For the most part, since synsanity is a subset of SYNPROXY, the performance benchmarks from this SYNPROXY slide deck (and other documentations) are still relevant: http://people.netfilter.org/hawk/presentations/devconf2014/i...

In general in the order of a hundred thousand PPS per core is easily possible, but it really depends how busy the server is, since it primarily comes down to how much CPU is being used to generate the syncookies instead of being allocated to the application.

Why don't you guys just run the most modern stable Kernel as opposed to 3.X?

You say: While Linux 4.x has a patch to send SYN cookies under a per-CPU-core socket lock, which does fix the problem, we wanted a solution that allowed us to use an existing, maintained kernel with upstream security patches. We didn’t want to roll and maintain an entire custom kernel and all related future security patches just to mitigate this form of attack. Patching Linux 3.x to backport the socket lock change was also a similar maintenance burden we wanted to avoid.

But, if this is the case, what's wrong with using Linux's LTS kernel, or the stable current kernel? The kernel.org team does a great job maintaining these. Which upstream vendor are you relying on?

Depending on your workload, and your kernel version, there are some fairly large performance improvements. These performance improvements in my testing primarily land in networking and I/O -- effecting many applications.

I think your work is awesome. I just think backporting, and not installing linux-image-generic might be a lot work for not much benefit. I'd love to hear your reasoning that leads you to believe that's a better option for security or stability compared to using the trusty kernel.

Also, typo: `causese all` -> `causes all`.

Great question! A large portion of our infrastructure still runs on Ubuntu Precise, and so the default "supported" options for the kernel on those machines are 3.2 or 3.13. Once you're on newer releases you get 4.x support, at the very least in backports, and at that point I would definitely agree the upgrade is worthwhile.

Have you thought of pulling in the latest Ubuntu HWE?

So instead of using a SYN attack, attackers can just run a full-open tcp attack. Not as effective if all you have is one or two servers with big pipes, but a botnet should make it feasible.

Reply on news.ycombinator.com