400Gbps: Winter of Whopping Weekend DDoS Attacks | Hacker News Reader

400Gbps: Winter of Whopping Weekend DDoS Attacks

blog.cloudflare.com

109 pointsriqbal61 comments

Too bad this was written as a Cloudflare advert, instead of emphasizing the changes that ISPs need to make (ingress source address verification) to make these attacks much harder.

https://www.ietf.org/rfc/rfc2827.txt

Egress address verification? Are you referring to RPF, or http://www.bcp38.info/index.php/Main_Page ?

uRPF is a challenge. In some cases, the consequence is flows must route symmetrically through a given network. That's not always the most efficient, flexible, or cost-effective way to go, particularly if you're a large provider sloshing massive amounts of traffic through your network.

smart edge, dumb core.

if edge networks implemented uRPF, it would put a huge dent in the ease of mounting DDoS attacks. unfortunately, the economics aren't there.

Agreed with your first point, but not sure about your second - how many of these attacks could really be dented with uRPF? My understanding is the majority are either distributed (coming from multiple ASes) or from compromised accounts in Azure, GC, or AWS.

If uRPF was stopping me from successfully attacking a destination, I would just not spoof. ACLs, uRPF, and anything stateful are still not going to 100% mitigate an attack from a broad range of addresses.

We've written about BCP 38 repeatedly:

https://blog.cloudflare.com/understanding-and-mitigating-ntp...

https://blog.cloudflare.com/technical-details-behind-a-400gb...

https://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-o...

uRPF is a pain to people who want to do any kind of creative TE with their network. For residential networks, I somewhat understand the value, but it makes dual homing unreasonable for anyone who doesn't have an ASN + BFP sessions.

I'd argue rather than this, we need to be able take Flowspec (RFC5575), and make it more reasonable for transit.

Good in theory - doesn't work in practice, at all, really. Imagine, lets say, still good MX960 with 3D 32x10GE line cards, now to apply mid size (10k lines) ACL on every port and see performance and stability. BCP38 in reality should be really-really on the edge of the consumer based ISPs, can't be applier to Tier1 and Tier2 ISPs where many believe it should be.

Maybe a little less "we are the best", "our hardware can handle nearly everything", "we can take it", etc. next time...

there is a reason some folks call them blogflare. They have great engineering talent and a marketing team that knows exactly how to acquire customers and more engineering talent.

and that's through their blog posts.

Actually we've only recently created a marketing team at CloudFlare and hired someone to run it. Most of the blog posts you see are generated organically by engineers or others who want are interested in writing about something they've worked on.

great engineers who can also articulate the technology in a polished, publishable form? not saying it's impossible, or that you didn't just setup a marketing team. but please pass my regards to whomever is editing before publishing. it's got a perfect, polished tone that any marketer would want.

Now that we have a marketing team we do have someone on that team who helps polish posts but that's a pretty new thing. In the past posts were polished by the likes of me or eastdakota. But that's assuming that they need polishing; one thing we look for in hiring is ability to explain.

A bit of feedback... I usually love your articles, but I really hated this one. It is just so blatantly self-promoting it turns me off. Just something to consider... Marketing can be overdone easily.

Fair enough. I guess we miss the mark from time to time.

Does every single cloudflare blog post need to appear on Hackernews?

"talent and a marketing team that knows exactly how to acquire customers and more engineering talent."

This resonated with me. I pursued a job opening there based on their blog posts and the engineering culture that was presented in them. I really liked the engineering folks I met at my onsite, they were all awesome. The recruiter on the other hand was a an idiot and probably one of the most unprofessional recruiters I have had to interact with.

I believe the intention of the blogs is in fact to attract engineering talent and that's fine. However it's a shame that the HR/recruiting part of that pipeline is the polar opposite of the thoughtfulness and thoroughness of their engineering culture - clueless and unprofessional.

Now a company is not allowed to be proud of their service? Cloudflare does a good job, and they're proud of their accomplishments, and they have a right to be. There's nothing wrong with that! I imagine you're also proud of your accomplishments.

You can be proud, but you can also overdo it. A man's praise in his own mouth stinks.

It would be much more sincere if cloudflare wasn't also hosting DDoS-for-hire services [0]. While I can understand the "free speech" argument, it still has a funny smell.

[0] http://krebsonsecurity.com/2014/12/lizard-kids-a-long-trail-...

Has there been any recent jump in the size or grade of bots available to botnet operators or is there some other technological reason underlying this change in magnitude?

It seems that the magnitude of monitored attacks is increased, not necessarily the original size. From the last paragraph:

> You might think that as CloudFlare grows, the relative scale of attacks will shrink in comparison to legitimate traffic. The funny thing is, the opposite is true. Every time we upgrade our network capacity, we see larger attacks because we don’t have to resort to blackholing. With more capacity, we’re able to withstand and accurately measure even bigger attacks.

I think its a good question to ask if the way in which attacks are generated these days has enabled greater size. Cloudflare points out that they may have already been this big, they just couldn't measure them before.

What I find most interesting is that this problem is providing a lot of pressure on ISPs to co-operate more. There are a lot of competing agendas, which we've seen with the very outsized fights of NetFlix against tier 1 and tier 2 transit providers, but those transit providers are being hurt by these people. By cooperating at some level it may be possible to mitigate the attack surface available to DDoS agents.

I always appreciate Cloudflare's write ups. That said, once folks are attacking it, and since you can measure things without black holing traffic, is it possible to identify machines participating ? Specifically to work with ISPs and network service vendors such that such attacks would serve to highlight compromised machines and then working with the ISPs of those compromised machines create an automated channel back to the operator?

It depends heavily on the type of attack. In many cases it is not like a syn flood which commonly used random spoofed (i.e. fake) source ip addresses.

With more and more network attacks and an increase in security breaches, I think we need to rethink some parts of the Internet for better security and reliability.

There's an interesting project going on - SAFENetwork[1] - similar to ipfs, which has the potential to render DDoS attacks obsolete for some use cases, if it gets adoption of course.

[1] https://safenetwork.org

Given the size of the attacks, shouldn't it be possible to identify who is behind these (or at least where it is coming from)? Or are they all distributed attacks from random hacked PCs? I find it hard to believe there are still enough vulnerable PCs: the count of those should be going down, not up. I guess if the source is nations the actually source might be difficult to pinpoint.

There are plenty of vulnerable PCs: http://krebsonsecurity.com/tag/botnet/

These volumetric attacks are likely UDP amplification attacks from hacked servers, not PC s.

They don't even have to be hacked, just expose vulnerable services.

I think OP is referring to the origin, not the mirror.

In the lab it's difficult to generate 400Gb/s traffic.

From my perspective it seems like this attack is pretty sophisticated and has access to some pretty fat pipes. It's hard to imagine that this scale of attack could come from some infected PCs.

Can an expert weigh in here on the difficulty of generating this much traffic in an attack?

I'm curious what happens if a free website is on the receiving end of a L3 attack, since only business plans and higher are advertised as having "Advanced DDoS support". Does CF dynamically adjust DNS and send the malicious traffic to your origin once it hits a certain threshold?

Matthew answered that below, but at this point our L3/L4 attack handling is mostly automatic and we would protect the FREE web site. We can dynamically adjust DNS to do all sorts of things when we feel like it.

Funny, stories like this to me always read as someone hacking themselves for job security; clearly not the case. Curious who the target was Cloudflare, or their clients.

The article mentions some of the recent attacks were on fairly benign websites, though that's subjective and could be construed as a sales pitch.

Missed that, thanks. Seems if they truly were websites of little note that the target was likely Cloudflair; my opinion, though guessing if they went down, the real attack would happen.

Targets are typically clients. Sometimes we can reasonable speculate as to motive, other times we're perplexed. One example: we saw a huge uptick in attacks targeting adoption agencies. Couldn't figure it out until we learned that adoption of Russian children by US/EU parents was a major hot button political issue in Russia. In other words, there's always a motive, it's just not always clear to us what it is.

Yeah, that's obvious to me that the adoption sites like that are targets, but see how it be bit puzzling off the cuff, or something of the sort. Not sure in fact what for sure would be a target that clearly lacks anything of note.

Usually forums of one sort or another. Obviously they're being attacked for something that's being posted there, but usually we have no idea exactly what or why.

I wish we could see an article like this for AWS.

I don't think AWS mitigates DDOS. If an app is under DDOS at AWS and was using an Elastic Load Balancer, the app would have to pay for the inbound bandwidth through the ELB and would have to pay for scale-up to respond to the increased traffic load.

It'd be much more cost effective to pay CloudFlare a fixed monthly fee and stop the DDOS attack before it even gets to your AWS resources.

The more I read from CF, the more I'm impressed and am definitely considering this.

Last time I looked at the AWS jobs site, they were hiring for a DDoS mitigation team.

having a slight understanding but no direct knowledge of their network infrastructure, that's probably for financial not technical reasons (they can handle the largest volumetric attacks and then some)

Agreed. We're full-stack AWS hosted, and I have no real indication of whether or not we've ever been attacked (doubtful) or if AWS could automatically mitigate an attack (possibly??). I presume Netflix is probably a common target, and they're AWS-heavy, but I have no way of knowing for sure.

> I presume Netflix is probably a common target, and they're AWS-heavy, but I have no way of knowing for sure.

Well, DDoS operators usually don't do their job for fun (okay, I'm discarding Lulzsec and other vandalism-motivated folks here).

It's much more profitable to hunt for not-CDN-backed sites and extort them for money. Or to look for processor-heavy API endpoints (e.g. login with huge work factors) to improve the DDoS effectivity, but then again a lot of DDoS ops are script-kiddies looking for a quick bunch of monies, all too often not skilled enough for this.

ELB is simply a Reverse Proxy with an upstream pointing to your app servers. Last time I checked, I think they were running nginx. I doubt they have advanced DDOS detection code running on that reverse proxy.

Netflix likely implements their own DDOS protection layer somewhere in their routing library. For example, since most resource-intensive Netflix endpoints are protected (you have to login to watch a movie), they could use rate-limiting based on User Account ID.

He last DDoS article from Amazon was a white paper which amounted to "we have enough capacity to let clients out-scale any attack." AKA it becomes a DDoS on your wallet instead of your server.

Edit: found it.

https://d0.awsstatic.com/whitepapers/DDoS_White_Paper_June20...

I run one of the largest legal bitcoin gambling sites (name in profile if interested) and was recently extorted for a whole week by DDoS's unless I paid (something that I never would do).

The specifics of our casino require us to have a pretty quality low-latency connection, so AWS was a first choice. I was recommended to use ELB which could mitigate the level3 attacks (primariy SYN floods we were seeing) and automatically scale.

I setup a test instances, and tried orchestrate my own SYN flood which it handled without issue (while a normal ec2 instance would just collapse). So I put production behind ELBs, and a few hours later the attacks started.

The first thing was that under stress ELB stops being able to make connections, and starts failing your origins for status check (even when they were getting no load), but was able to successfully scale to mitigate the attack within 5-7 minutes. The attacker tried again a few hours later, but it seems things were nice and pre-warmed and the attack didn't get far.

For about a day there was a nice calm, and ELBs was performing absolutely fantastic (in terms of latency/jitter), and then I got the email that he's about to throw everything at me. And within a few minutes, he took out all the ELBs and it wasn't able to scale to handle the traffic for over an hour this time.

During this time I was in contact with CF's excellent support, and they offered me protection for websockets. It has slightly more latency than ELBs, but is actually an anti-ddos service, and works wonderfully. Since being fully behind CF we've only had one DDoS related outage, and it was specific to Seoul for about ~3 or 4 hours.

Can't recommend them enough, quite literally my business might not be around if it wasn't for them.

https://www.youtube.com/watch?v=OT2y3DzMEmQ

Who's behind these?

Dejavu.

"the largest distributed denial of service (DDoS) attacks ever seen" ... returns again part 2

Hmm Looks like if you want to benefit from this level 3 DDos protection, you'll need to hand $200 / month over to Cloudflare.

Sounds fishy to me.

Actually, in the case of the DDoS that is primarily the focus of this post, the site being attacked used the free version of our service. Volumetric attacks have become automated enough to deal with that we find the benefits of stopping them, even for free customers, and recording the resources used by the attackers, is well worth it. Plus we generally believe you shouldn't have to pay in order to be safe from bullies online. Our free level of DDoS mitigation today is better than what you could have paid for, at any cost, from any provider 18 months ago.

$200/month feels cheap to me considering how much effort and money it would take to get this defense if you are on your own.

Just talking about the bandwidth cost, if it was even possible for me to get the 400gbit/s bandwidth required to weather such an attack, it would cost me way more than $200 per day with my current ISP. And that doesn't yet include my own time and stress.

If the stuff we're running was valuable enough to DOS, I would totally consider the 200/month extra

If internet connection speeds continue improve, it is fairly obvious DDOS traffic will increase. Claiming that it is bigger than ever when Internet connection speeds are higher than ever is like claiming that gravity from the Sun is stronger when you are closer to the Sun.

Pretty sure they are saying that because they can handle more traffic they can measure the attack now instead of blackholing...

Reply on news.ycombinator.com