Hacker News Reader

Comment by wesleytodd | Hacker News Reader

My startup's site gets DDOS'd about once a week. We have seen a huge range of attacks from UDP floods, to wordpress pingback attacks, to directed attacks on our services.

We have many layers of protection:

* We run iptables and an api we wrote on our ingest servers. We run failtoban on a separate set of servers. When fail2ban sees something, we have it hit the api and add the iptables rules. This offloads the cpu of failtoban from our ingest servers.

* We block groups of known hosting company IP blocks, like digital ocean and linode. These were common sources of attacks.

* Our services all have rate limits which we throttle based on IP

* We have monitoring and auto-scaling which responds pretty quickly when needed. And has service level granularity.

* Recently moved behind cloudflare because google cloud did not protect us from attacks like the UDP floods which didn't even reach our servers.

EDIT: formatting

View on HN

One other thing to add:

If they attackers are persistent, there is really no way to guarantee zero down time. THEY WILL FIND A WAY. Just make sure your stake holders know you are doing everything in your power to resolve the issues, and then actually do those things.

An anecdote:

We had been seeing DDOS attacks for a few weeks, so we had most everything locked down and working. But then suddenly one of the most important parts of our site started going down under load. That part is a real time chat system. We looked for which chat room had the load and it was one which did not require a user be registered. We switched the room into registered users only mode and thought we had solved it.

About 5 minutes later the attack came back with all registered users. We were amazed, becuase there is no way the attackers could have registered that many accounts in 5 minutes because of our rate limiting on that. Turns out that they had spend the past week or so registering users in case they needed them :)

Great tips and examples. Makes me wonder what site/service you work on that attracts attackers like that.

https://www.stream.me/

We have some controversial users...

I guess some people just really hate kittens :)

https://www.stream.me/kittendorm

Interesting service. Yeah, I could see you people getting attention. Love the homepage pic haha.

Cloudflare doesn't help you against UDP floods if your backend is publicly accessible on the internet.

For example:

curl https://104.154.116.193 -H 'Host: www.stream.me' -v -k

Indeed, it's a recent integration and we haven't yet shut off that IP.

Btw, check out curl's --resolve flag. You can use it to override default DNS resolution and can then drop the -k flag.

If I remember correctly, Digital Ocean was by far the greatest source of Wordpress pingbacks. They've got a severe problem on their hands. I had submitted a report to their abuse contact weeks (months?) but have not yet heard back.

Reply on news.ycombinator.com