Comment by tzs | Hacker News Reader

Comment by tzs | Hacker News Reader

Parent Full thread

> It's unclear if the CTO just doesn't understand that "DDoS" generally implies malice, or if they're intentionally using that language to blame users for using their product.

I've definitely seen "DDoS" used when there was no malice, such as when a developer accidentally releases a client that generates way more traffic than it was supposed to. Probably because we don't seem to have a good term for "event that at the server looks exactly like a malicious DDoS attack but was actually due to a mistake or to the server becoming unexpectedly popular" :-).

My favorite example of whatever we are supposed to call this was John Carmack in 1997. From his 1997-12-09 .plan:

> Cyrix has a new processor that is significantly faster at single precision floating point calculations if you don't do any double precision calculations anywhere.

> Quake had always kept its timebase as a double precision seconds value, but I agreed to change it over to an integer millisecond timer to allow the global setting of single precision mode.

> We went through and changed all the uses of it that we found, but the routine that sends heartbeats to the master servers was missed.

> So, instead of sending a packet every 300 seconds, it is sending one every 300 MILLISECONDS.

> Oops.

> To a server, it won't really make a difference. A tiny extra packet three times a second is a fraction of the bandwidth of a player.

> However, if there are thousands of network games in progress, that is a LOT of packets flooding idsoftware.com.

> So, please download the new executable if you are going to run any servers (even servers started through the menus).

That's fair. Maybe my security background is shining through here. I guess we used to have "slashdotting" but that doesn't generalize well :)

I did do some napkin math to quantify how much that bad traffic may have been: HA estimates between 6857-25576 intallations of the MyQ integration. Let's say 16k clients. HA makes it really easy to detect and "add" the integration (which counts as an installation even if it's not configured), so, that's definitely not all clients hitting the API. Let's say it's 50%, so 8k actually using it. Most users just notice myQ is broken. Let's say some fraction retry, which would look the same as an extra user from a volume perspective. Call it an even 10k users (including repeat users).

The most recent change is after they broke everything past the OAuth dance. Let's say the OAuth request is 1kB. The retry code retries up to 5 times with exponential backoff. Let's say 5 requests over 10 min.

(5 requests / 10 minutes) * 1 request/user * 10k users = 5k requests/minute, or 83 per second, amounting to 83kB/s inbound.

There's no reason to assume those requests would synchronize, but I'm sure there's something (let's say every single myQ user updated at the same time).

If what they're saying is true, sounds like actually malicious botnet wielders can ransom the living daylights out of them. Given 1Tbs DDoS attacks they'd only need a tiny fraction of the full bore ion cannon! ;-)

[1]: https://github.com/arraylabs/pymyq/blob/master/pymyq/request...

83 rps would be a challenge when hitting a Java EE app written to make use of tutorial-level ORM code without any caching or optimizations. An app where a request takes 300ms to resolve (pulling numbers out of hat for an average poorly written Java EE app; ignorantly assuming 300 ms are spent with 100% CPU utilization of a single core), would require a 24-core machine to keep up with 83 rps. Accounting for some peaks in usage (how about 5x around 7-8am?), 400 rps could make almost every morning an "all hands on deck" event for the ops?

A term I hear a lot for non-malicious or non-intentional DDOS is the Hug of death.

> I've definitely seen "DDoS" used when there was no malice, such as when a developer accidentally releases a client that generates way more traffic than it was supposed to. Probably because we don't seem to have a good term for "event that at the server looks exactly like a malicious DDoS attack but was actually due to a mistake or to the server becoming unexpectedly popular" :-).

This is a problem with the service, not with the developer.

If the service (doesn't want) / (can't handle) something, then it should rate limit it's response.

If the service can't handle "0.2%" of it's clients making a 'not unreasonable' amount of requests, how will the service hold up against a hostile actor who aims to DDOS their service.

> I've definitely seen "DDoS" used when there was no malice,

Absolutely. Used to work on the Identity team somewhere. Dev accidentally removed code that was supposed to cache a token on a very chatty service. Brought auth to its knees and called it DDoS.

Reply on news.ycombinator.com