Jerks on the Internet: what my first DDoS taught me
sergiomattei.com
sergiomattei.com
For some reason, this isn't done much. I have it on a site of mine. I didn't notice for a week that someone was making a huge number of requests and not even waiting for the task to complete. It didn't hurt anything.
There is no 100% secure system.
If however your DOS attack is an attacker making lower-volume CPU-expensive requests on your site, there's plenty of things to help mitigate the assault.
Sadly DDoS has at times been used as a blanket term that also includes DoS.
https://www.cloudflare.com/learning/ddos/famous-ddos-attacks...
Certainly no need for cloudflare
1) the three most important metrics for any endpoint are error rate, lantency, throughput. So i hope you've learned to not be surprised that abnormal throughput (either via ddos attacks or friendly n+1 queries) is a common error condition.
2) banning ip addresses is useless and often counter productive. If possible, short-circuit requests from an ip so you can isolate them and then determine how much damaging knowledge they know (while letting them think they're still hitting your real service)
3) feature development is a great goal, but don't forget that the most important feature is availability. Spending an extra 30 minutes to consider things like pagination and so on, end up being worth it if you think you might get attacked more than 0 times per year.
What does that mean?
Adding pagination seems reasonable, but it may have broken clients who didn't expect pagination to be there.
curl -k https://134.209.46.107/products/ -H "Host: api.getmakerlog.com"That pretty much describes the whole history of internet.
No caching?
Since they're already using Nginx, if they don't want to bother with learning anything else, it's a couple of hours of research to learn how to set up rock solid basic caching using Nginx. It'll quickly get you 85% of the way on caching, until you need something better. Set Nginx loose to do one of the things it's very good at.
There's not much you can achieve for your own gains with a DDoS. I've heard of rare cases of extortion or underhanded business practices to hurt competitors.
Well, no. If you _only_ liked sparks and glass shards flying all around - you'd buy yourself a car and destroyed it yourself.
Destroying the car of a neighbour implies something going between you and the neighbour.
I guess humans are closer to monkeys than we like to believe and still like throwing poop at each other.
Don't do that
>I trust my users completely.
Don't do that
>Prioritize bugfixes over new features
while that is a nice thought, it is unlikely to be so simply followed. "the road to hell is paved with good intentions"
>people editing other’s tasks for example, haha
That's a very cavelier attitude to take. Quite frankly that should have been baked in from the get go with tests to verify.
You see this situation as someone being a jerk. Someone could have accidentally done the same thing due to the lack of planning on multiple levels.
It was patched though and there was no evidence that anyone ever used it.
Most of the DDoS providers in this space are insanely expensive so I’m very glad Cloudflare has existed!
Let me just say, that's a really dumb DDOS attack.
Edit: one other thing, we spell it psych and not sike.
I have been informed by someone a generation younger than me that "the kids" are intentionally spelling it "sike" these days.
Wikionary lists it as a variant of "psych."
I confess I did have to google how to spell 'psych' properly. started from 'pysch'.
But the topic recently came up in conversation with my teenaged niece, she claims it's definitely spelled "sike" even though she's fully aware of the etymology.
I chocked it up to a "kids these days" generational kinda thing, like how we got "phat" in the 70s and "kewl" in the 90s, both of which are now in the Oxford English Dictionary.
chalked it up ;)
2. Your clients should not be the first one to alert your site is down. Pingdom does a great job to alert you before your customers/users do.
3. The author brought up some of the queries weren't paginated and running an expensive SQL query, so there's a few options. Since he's using Django there's some Django specific options in this list.
A) Implement a backend cache that will return back the JSON query (throw it on a redis). Cache and return that from the backend.
B) Add a Django throttle to the view (can be done via IP / username).
C) Enable logged in users only to access endpoint (harder to do on the fly though, since you need to make changes to your frontend). If a logged in user is causing you hell, turn off signups and kick that user off.
D) Have CloudFlare cache a public response for you on endpoints and return it (you need to make sure the JSON should always be the same for every API call though, which is very very risky).
E) Author brought up DRF JSON serialization is slow. Another alternative is to use Serpy which sees a 50-100x speedup. I'd only recommend that for complex JSON payloads. Not because it's hard, but because it's additional complexity.
The author is also using Dokku which is fine for most projects, but you'd imagine at some point it'll probably be switched onto a load balancer + web machines. Alerting can also be set on the load balancer level if it goes above % threshold.
Since he's using Dokku (so by that definition docker), they could probably use a log aggregation service that would allow him to access his logs much faster to see what's going on. Papertrail, etc.
Monitoring CPU usage would also be helpful here, but I'm not sure if Dokku allows that.
I couldn't figure out what's going on with my laptop, because the inspector short-circuited this goofy behavior. On a larger screen, I notice that the ".card-content" div has a ridiculous 150px of padding. It is nested in a ".card.blog-content-card" div, which has an atrocious 200px of margin. That in turn is nested in a ".blog-post-container.container" div, which has a merely unseemly 93px margin.
After 886px is used for white space, there's not much screen left for text. Might want to fix that?
If your gateway/webserver is running in a Docker container and you've published port 80/443, Docker will set up it's own IPTables rules, bypassing anything you've set up using UFW.
I have never been ddos's and all I ever receive are failed ssh attempts with simple passwords. Pretty much the easiest thing to tackle. But I'd love to know from DDOS'd people how their attacks looked?
From cloudflare logs all I see is a single IP address being blocked (multiple times? Or is it their multiple actions being blocked?).
EDIT: Thanks all, these were helpful answers.!
It started as a TCP SYN flood, which I had never seen before -- but since it originated from only two addresses, I could block them. Once I did, more addresses joined in on the attack. I figured out what was happening and how to prevent the connection table from filling up completely under the SYN flood, and then the attack changed shape into a UDP flood, from yet more addresses.
(Many of the addresses corresponded to free shell hosts and managed webhosts with easily exploitable PHP scripts. I tried to inform the people running those hosts that they were being used for an attack, but the vast majority of them seemed to ignore that. A couple of people responded and we could figure out kinda what had happened and what scripts were involved. Remember, kids, that connecting a machine to the internet is a great responsibility.)
Once you're flooded and your process load spikes, you can't see what your web server has run out of workers to handle.
Netstat will give you a better picture as far as IPs/connections once you get the process load down so you can actually run anything.
Layer 7 attacks are different; you can't spoof those, so you don't get perfect distribution -- but there are lots of ways to distribute simple requests. If the requests are coming from a botnet, there's usually a lot of control about what the requests look like, but if they're coming in through tricking other software (which is unfortunately common), then at least you'll likely have some identifying information; it's dumb to block things by user-agent, but it can be pretty effective. The way to handle these is really to try to make sure the effort your server spends is roughly on par with the effort the client spends; and try to make sure you're running the best optimized TLS handshakes you can (ECC certs are easier on servers than RSA).
I was also attacked by a single ip that sent 10gbit of nonsense to apache, I had to contact my isp to get them to block the ip down stream from me, if they had used a bot net I don't think I would have been able to stop it.