The largest DDoS attack to date, peaking above 398M rps
cloud.google.com
cloud.google.com
The novel HTTP/2 'Rapid Reset' DDoS attack - https://news.ycombinator.com/item?id=37830987
HTTP/2 Zero-Day Vulnerability Results in Record-Breaking DDoS Attacks - https://news.ycombinator.com/item?id=37830998
The hard part (and it truly is hard!) is convincing a few companies to do this. It risks user complaints in the short term, to solve a problem that may not be very acute for the largest companies (who can simply absorb these attacks).
(No I don’t expect any response but I am just leaving this thought for those who stumble on this thread in the future).
ISPs do this literally all the time. They sell services that do this.
not necessarily true
Shout out to Dutch ISP XS4ALL who was (is?) very very strict and active in this space.
Also, sounds illegal.
Step 2: Execute DDoS
Step 3: Prove to others you are responsible by using private key
(the /s is just on the "oh wait" part, not the whole post)
It's not obvious what's the value of having the largest ineffective attack.
One gets you more money in the short term. The other one gets you more street cred - which gets you more money in the long term.
If they're advanced, they are doing it to test capabilities and responses. The Taliban used to pay kids to light off firecrackers outside base to check defensive TTPs. It also had the effect of desensitizing the sound of gunfire.
Really good adversaries know how to accomplish the latter while appearing as the former.
Bring back CotDC, MoD, even lulzsec. I miss the days of the internet being the open seas and everyone having their own fun on it, user beware.
They then use their street cred to get paid by less scrupulous actors to attack their rivals. Sometimes the people paying are governments, sometimes just shady companies. For example last year there was a lot of crypto companies attacking each other's websites.
Most of the people who do this have a lot of technical skill but not a lot of opportunity to get paid for it based on where they live or the circumstances of their upbringing.
Now we need the SEO content side: "How we hit Google with 398M RPS".
"... you can do this manually, but our product makes it as easy as a sign up and API call. Talk to us about pricing. [Python API example].
What I'm not sure of is why Google published this. I can't figure out what their strategy is here. We never published about the attacks we absorbed because we didn't want them to know our capabilities.
Unless this is marketing for Google Cloud?
If you read the article, there are plenty of marketing remarks in there to get you to use Google Cloud
That seems likely here if they're claiming this is the largest DDOS ever.
Besides publicity, there is also link to a list of advisories that may be of interest to other cloud operators and users.
We? Netflix or Reddit? I know for a fact that Amazon doesn't.
At eBay/PayPal we filed patents on our DDOS shield, since it was as far as we knew the first one to exist, but that was about the only public information on it.
At reddit and Netflix we didn't actually have to deal with it because AWS just absorbed (or mitigated) it before it ever hit us. We only had to deal with L7 attacks, which we had shields in place for.
As for why to write about it, it's a new type of attack that resulted in almost an order of magnitude increase in attack size. That's interesting and newsworthy by itself, and publishing a concrete number gives people an idea of the size of the problem and the trendlines.
This is also something that needed a CVE, so it was going to be very public anyway. If nothing is written about it, at a minimum Cloud customers will be flooding their support reps with questions about whether the vulnerability applies to them.
Pretty clever
How is this an advantage? Can someone explain please?
You could add some smarts to the server or reverse proxy that delays starting work in case a cancellation request quickly arrives. This is probably part of the mitigation work they refer to in the article.
They point out in the article that it's a better practice to immediately limit stream creation when you detect abuse - not wait for a round trip to complete first. I'm sure there's a good reason for the original guidelines; I'm just trying to get it and haven't found anything clarifying through Google. Was it specified before the rise of modern attacks?
After all you could say any client ignoring a GOAWAY is either bugged or malicious but certainly not until you get confirmation they go it.
Here with the "attack", it's simply exploiting the ability of HTTP/2 to compress requests and reduce them to just a few bytes, meaning that within a few kilobytes of data you can easily have hundreds of requests. Again this is not new and was already being discussed in 2012 about SPDY's use of zlib to compress requests.
The extra stuff that seems to have made this attack "new" for such service providers is that attackers took care of closing their requests so as not to have to wait for a response and be able to fill the wire with a flow of request. Again this has been known from the inception of HTTP/2 and routinely met by those dealing with proxies which timeout and send headers followed by rst_stream.
Here it makes noise because new records were broken, and likely because the stacks in place were not properly designed so they omitted to check for the real number of streams and only focused on the protocol validity...
LOL. No there are plenty of legitimate enterprises as well as opportunity to immigrate. Especially in tech. These guys are just criminals.
I know people who can't relocate because of communication issues and/or cultural differences.
No they aren't criminals, but they are definitely underpaid compared to those who managed to relocate.
Blaming it on the people that knock them off will not make improve the situation.
Also the participants are sometimes innocently recruited victims for the attack. I blame app insecure defaults.
The trend since 2015 is to get worst as you will see in the bottom layer of this graph: https://www.digitalattackmap.com/
> Who has an incentive to carry out these DDos attacks?
Did you read the comment I was replying to?
Sitting at the end of whatever network, you will not be able to do anything against a sufficient volume attack.
With the Mirai botnet, some of the creators had a DDOS mitigation company as well: they'd sell one party the weapon, and sell another party the defense against that weapon.
Sometimes it's for the street cred, or the lulz, or just the challenge of building a botnet.
Seen it happen with big DDoS on clients. Furaffinity, one of the larger furry webistes, and a constant drama magnet, was a client at a former job. They got DDoS'd hard, and in between scripted DDoS hits they slammed the hell out of their web applications to get vulns and do credential stuffing.
As in blast em, lighten it up just enough to get a ssh or nmap through a few times, blast em again, and repeat until they got in.
Is also why you want out-of-band solution that doesn't touch your infra much.
I've heard this before, but I still don't understand what purpose the DDoS serves here. Distraction, so the actual attack drowns in the noise?
If you analyze the situation from the perspective of "Who benefits from it?", then the answer is clearly: Google benefits from it (they are so good, they can mitigate gigantic DDoS attacks). So, I don't think it's that crazy to think this is all a publicity stunt .
"With or without patches, organizations would need to make significant infrastructure investments to keep services running in the face of attacks of any moderate size and larger. Instead of bearing that expense themselves, organizations running services on Google Cloud can take advantage of our investment in capacity at global scale in our Cross-Cloud Network to deliver and protect their applications."
I'm pretty sure someone will find a way to take down GOOG/AWS/Azure/etc through a DDoS so large nothing will work for anyone.
Anyone involved in these type of attacks (at internet-infrastructure scale or targeting specific companies) brave/crazy enough to create a throwaway account and tell hn about the motivations?
We also had some actual attacks so we made a system that detect anomalies (like more than 50rps per IP) and raises alert.
...which was thwarted by frontend developers again, as they loaded few hundred tiny icons at once that triggered that alert routinely, and only thru http2 multiplexing their idiotic design patents haven't bitten them before.
Performance metrics were taking a dive and triggered automated alerts - average response time jumped from 200ms to 2000, 8000, and eventually began approaching 15000, at which point requests were timing out all over the place. At first I was wondering if my recently-deployed MR was responsible, but upon further investigation, it was from one developer on another team doing two very dumb things with their application prototype:
- Constantly retrying the same query with no filtering and setting the maximum allowed page size
- Sending massive quantities of small queries for specific individuals (50-100 per second) which quickly hit the rate limit, and immediately resuming after the rate limit expired
Some developer outreach was required...
You might not remember this, but before Google, paid search placement was par for the course. One of Google's innovations, one of the things that really endeared it to users was clearly labeling their ads.
So, yes - it was inevitable. And, in fact, Google probably staved it off at least a decade; maybe more.
My bet is on the latter. Enshitification is a direct product of greed. No crafts person or creator I know of goes into something they enjoy creating with the intent to make it this monstrosity of money extraction. Most creators have a drive for their creation to be shared and experienced by many.
Yes, you may want to get a reward in the process and for some creators, their motives may change over time if they see an opportunity to turn their creation into a wealth machine for themselves so they can do whatever after.
Enshitification I believe is a secondary effect of something that becomes successful for the owners of something and either they or others change motives towards value. Optimization is no longer about the creation, sharing, experience, humanitarian, whatever motive and shifts to money. The second that becomes the goal, enshitication is just part of the optimization journey. In my line of thinking, it's the same reason monopolies or near monopolies tend to form, these are merely further states along optimization strategies in the monetary/wealth extraction goal.
Part of that process is that when something starts to succeed, it attracts people with these goals so the goal of something shifts pretty rapidly.
The drive to create the thing comes from sincere enthusiasm + excitement that simply don’t leave me alone.
Shifting the thing’s purpose from the original one that filled me with enthusiasm to the purpose of simply extracting money, corrupts its original purpose.
I call this corruption.
it is my observation that one kind of people create, and another kind of people corrupt.
Unfortunately the economic system we have accepted as normal, leans toward corruption and extraction.
It's not surprising Google search is now crap, it's what happened to all the old search engines. It's only surprising that it took 15-20 years (depending on perspective), and in the mean time, they've developed a big ecosystem of other stuff.
This kind of attack is nothing like the actual DDoS attacks, but it's a lot more common in my experience, but also relatively easy to mitigate with something like Cloudflare or Akamai (which is what I'd recommend to my customers).
TL;DR: HTTP/2 is internally concurrent, can handle multiple streams. It is possible in HTTP/2 to send a nasty request that looks like so:
- GET x1
- GET x2
- GET x3
- ...
- GET x100
- Actually, cancel all of the above (uses multiple RST_STREAM frames)
- GET x101
- GET x102
- (...)
- GET x200
- Actually, cancel all of the above (uses multiple RST_STREAM frames)
- (...)
Maybe it's just me, but it's always fun to see such creative and simple abuses of protocols/code.
Curious if there's anyone in the HN crowd that works at this level in one of the major vendors. What happens during an attack of this scale? Are there people from Cloudflare + Google + AWS on a live videoconference call co-ordinating with each other in real-time to mitigate it? Or is each vendor mostly observing from a distance what is happening elsewhere, and solely focussed on sorting their own problems out?
Is it only about having a large enough ingress pipe that you can weather however many Gb/s you are being bombarded with, and still having some spare capacity for legitimate traffic?
In this case, it might mean recognizing when a client rapidly resets streams, and either moving that traffic to a slow lane or filtering it entirely.
You configure your server to drop all traffic which wasn't set by cloudflare, which is efficient.
Not all ddos attacks are based on volume though, some are exploiting native features of a protocol, like the slow loris attack
https://www.cloudflare.com/learning/ddos/ddos-attack-tools/s...
The way we used to do it is have "filter boxes" with a real anycast IP address's which reverse connect to your origin.
This helps a lot because it keeps a lot of traffic localised instead of allowing it to collect in one place. Anycast should also mean you have a failover mechanism; but if it fails then you're only down in one section of the world where the most bots are anyway, which is usually not as bad as being down globally.
b) ability to filter the noise from real traffic as far as possible (i.e. there is little point in taking in a big pipe of DDoS traffic and then just proxying it to the thinner pipe to the real backend - but if you can identify bad traffic you can drop it and not pass it through).
c) being a CDN helps as a side-effect (what the CDN serves doesn't load the backend services, what can be served from the CDN works for users even if the backend is slow or down)
A lot of DDoS traffic isn't actual HTTP traffic, it can be garbage targetted at your IP address to "fill the pipes" (bigger pipes help, as well as having multiple server geographically distributed). Some can be TCP SYN flood, to just open TCP connections and exhaust available ports. Etc. Oftentimes, multiple simple reverse proxies can handle these malformed requests in front of your server.
Then, for the most sophisticated queries that send seemingly-legitimate HTTP traffic, one has to handle them... It could be serving requests from a cache, adding captchas to slow attackers and identify legitimate traffic, enforcing rate limits, etc. Usually, you'd like to be able to tell if a request is legitimate or not before forwarding it to the actual server, and you can deploy all sorts of tools to do so.
I don't think this is right. If you have a meaningful amount of bandwidth, dynamically scaling it is getting a connection upgraded in weeks instead of months. If you don't have a meaningful amount of bandwidth, you're rely on your provider(s) to have enough bandwidth and again, they can't expand quickly.
> Some can be TCP SYN flood, to just open TCP connections and exhaust available ports.
If you have a tcp stack from maybe 2003 or later (so excluding macos, unless they changed something in the past four years), it will have synflood protection, with syncookies. In the event of a heavy synflood, your system will send at most one syn+ack per incoming syn, and actually accept connections on the incoming ack. Yes, you miss out on detailed tcp options, but it's not that big of a deal, unless the volume impacts your available bandwidth.
Also, as a tcp server, you can't meaningfuly run out of ports; your one listen ip:port can connect to all ip:ports, if you have the memory for it. You'll probably run out of total accepted sockets, but there's no real resource limit on partially accepted connections, because of syncookies. It can be much more draining when DDoS clients actually hold connections. But it's often simply about volumetrics, and it's easier to generate a high volume of SYN packets than to hold a connection.
In general, Google's internal cross-DC traffic is so much larger than anything anyone could DDoS them with that they can always find a way to deal with it.
https://blog.cloudflare.com/zero-day-rapid-reset-http2-recor...
"Furthermore, one crucial thing to note about the record-breaking attack is that it involved a modestly-sized botnet, consisting of roughly 20,000 machines"
(this assumes that 1 person = 1 device; some people share devices, most people have more than one, e.g. I have a laptop and a router, many people also have a phone, a work laptop, and whatnot – the average is probably >1, maybe even >2)
And lots of devices are not even that, Internet of Shit garbage is well known for being botnet-central, servers running insecure services are good fodder (it's very common for vulnerabilities to be used not to exploit the machine itself, but to install C&C and leverage the machine into other attacks), ...
An actual nation state would have tested it fully internally and started on the public internet at a scale bigger than 20,000 machines.
All the big providers could do this, just inject a little turnstile like page in front of the next cloudflare site you visit.
I would love to know if there's a compromised device on my network, and I don't have any real monitoring set up to detect it.
It's not a full solution, but at least informing users there is a problem is a good start.
Oh good. We can go back to the pre-HTTPS days where ISPs injected ads into HTML. Except this time we normalise it for the CDN provider.
I honestly don't see any other options, but I'd really wish for them to come through some community coordinated list of botnet infected IPs or something.
Also, stupid question from someone not that familiar with DDoS, can't you flood the target with requests even if the source address will be rejected? Or even if the IP packet has a falsified source address?
What if Google and Cloudflare collectively reverse-DoSed all the infected IPs, not by sending them any traffic, but simply by refusing to accept any connections from them to any part of their infrastructure?
Whoever is on those IPs will suddenly find that half the internet doesn't work anymore. Which is probably a good enough incentive for them to replace their router, format their PC, or whatever else is necessary to disinfect themselves.
In many parts of the world, landline IP allocations tend to be stable enough for this to have a real effect. Phones are a different story, but phones are also much less likely to be useful in a DDoS botnet. (The owner would immediately notice the sudden heat and data usage.)
If we're going to live in a world where a small number of companies own half the internet, at least they could use their power to do some good.
And then you have to fill in a new captcha every 5 minutes or so just to keep using google maps/gmail/search.
It's kinda annoying, and usually the culprit is someone else who shares my IP, not me (ie. a school, university, workplace, open wifi).
Plz fix.
And what about CGNAT?
People have been begging ISPs for ages to do a bit of egress filtering, for example, to prevent source address falsification. They've demonstrated time and again that they don't give a crap unless it affects their bottom line.
Your droplet suddenly tries to log into somebody else's server 10 times a second. The target of the attack complains to DigitalOcean, "hey, one of your customers is trying to hack me!" and attaches a log of the login attempts. DigitalOcean assumes that the report was made in good faith, forwards it to you and immediately suspends your droplet. It won't be reactivated until you reply with evidence that you have at least tried to clean up the problem. If it happens again, you won't get off so easily.
I suppose that a similar system, in a more real-time fashion, could be set up between the maintainers of the blacklist (Google, Cloudflare, Amazon, etc.) and the ISPs. No need for the ISPs to sniff on everyone's traffic if they can rely on good-faith reports from the lion's mouth that somebody from port 52384 on 11.22.33.44 is DDoSing a Google property. Even with CGNAT, the port will identify the customer responsible.
Because they probably don't care.
Here is one that I'm aware of: https://named-data.net
Independent of police, in bad communities your neighbors are willing to break in. In good communities they don't.
All large sites regularly get attacked.
The average skiddie’s motivations are that they’re bored. So they DoS a site they use regularly just to see.
Heck they generally don’t even mean to cause damage per-se, and just think it’s a funny use of their evening.
You have to stop thinking DoS attacks are always particularly personal. They really often just aren’t, and it’s a monumental pain in the ass to be on the receiving end.
Well used, but never attacked.
For this particular scenario, the public Internet would get so bad ("enshitified") that people would tend to leave it alone. For essential public services, governments would set up their own networks disconnected from the Internet, where all devices and their connections must be authenticated to a person or corporation[^1]. Maybe something equivalent would exist for corporations and to enable e-commerce.
[^1] China works like this already, to a high degree.
> Having security and trust be “intrinsic” to the network will require core layers to carry metadata about the users, applications and services being transported. If users need to register in order to have packets sent to their destination, the result is that network operators, and those who license the operators, can remove individual users’ access at any time.
https://dnsrf.org/.k-media/d3c1d810de1e98bdf7af7aa52406e837.... (critical of the proposal)
Just FYI: hetzner has free DDoS https://www.hetzner.com/unternehmen/ddos-schutz
I'm sure other hosting companies also offers it.
> In this final layer, we filter out attacks in the form of SYN floods, DNS floods, and invalid packets. We are also able to flexibly adapt to other unique attacks and to reliably mitigate them.
Which means any legit http2 connection will go just fine.
Even if such connection now triggers hundreds of substreams.
Push for end to end encrypted internet also means you can't really stop any more advanced attack. You could have just few dozen of hosts doing 20-30 connections each (i.e. "looking perfectly normal" for DDoS protection provider) generating tens of thousands per second in http2 streams.
I'm speaking from experience of mitigating attack like this. Our DDoS provider was near-useless..
If your computer is infected, I don't want to talk to you for a month. If it continues to be infected, I might up that to a year, or permanently ban you.
It's your problem. Go fix it.
(not to mention the number of false positives you'd get, etc etc)
> We might fix this, we might not, you DONT have a choice.
> Thank you for your continued business.
(swap in any corpo-service provider you personally like the most)
Blanket banning subnet ranges from services because of the actions of someone else is 3rd world shit.
One of the things that gets lost in this kind of debate is that the vast, vast majority of Internet users are not experts in how the Internet, computers, or their phones work. So expecting them to be able to "just not get exploited" is a naive strategy and bringing the pain to the ISP feels counterproductive because what, realistically, can they do to stop all of their unsophisticated users from getting themselves exploited?
At the end of the day, the vast majority of the users of the Internet do not care how it works - they want their email, they want their cat videos, and they want to check up on their high school ex on Facebook. How can we rearchitect the Internet to be a) open b) privacy protecting, and c) robust against these kinds of attacks so that the targets of DDOS attacks have better protection than paying a third party and hoping that that third party can protect them?
And yes, we can block entire subnets. You own the IP addresses, you're responsible for stuff coming out of them, at least to the degree that it's not maliscious to the web as a whole. (but not the content itself, of course)
I'm calling bullshit on these assumptions. The internet is a communications tool. If it's not communicating, it's broken. If you provide dynamic IPs to clients that attack people, you're breaking it. It's not my problem or something I should ever be expected to pay for.
To be clear, my point is that we're suggesting yet another layer of commercial, paid crap on top of a broken system in order to fix it. It'd be phenomenally better just to publicly identify place and methods where it's broken and let other folks with more vested interests than information consumers worry about it. Hell, I'm not interested in paying for the current busload of bytes I'm currently consuming for every one sentence of value I receive.
Every reasonable connectivity provider would pay attention to this info, or face intense complaints from its users with shared and dynamic IPs. It would identify sources of attacks, and block them at higher granularity level, reporting that the range has been cleared. (If a provider lied, everyone would stop believing it, and the disgruntled customers would leave it.)
For shared hosting providers it would mean blocking specific user accounts using a firewall, notifying users, and maybe even selling cleanup services.
For home internet users, it also would mean blocking specific users, contacting them, helping them identify the infected machine at home.
It would massively drive patching of old router firmware which is often cracked and infected. Same for IoT stuff, infected PCs, malicious apps on phones, etc. There would be an incentive to stay clean.
I would greatly prefer not having my semi-randomized IP blocked because someone used it maliciously a year ago.
If anybody is suggesting permanent bans of IPs, it's not me, at least not at a public level. I may very well choose privately to do that.
To clarify, I, personally chooses a blacklist policy. Not some other org. I think if you offload this onto any kind of external structure, it breaks again.
ADD: We make publicly-available, second-by-second, how the internet is broken and invite all comers, including me and my blocklist, to help fix it.
There's a huge commerical interest in NOT fixing the problem of random crap showing up, from dancing cats selling things to targeted inserted ads. I get it. We saw this same thing happen with adblockers. It's now going on with "free" VPNs. Can't fight that perverse incentive, so don't fight it.
The problem is that ISPs whose customers are originating the attacks from don’t give a shit.
If we have to give up 1% of legitimate traffic to thwart 90% of attacks, it is a good deal.
If you and other customers complain to your ISP (or switch), eventually they’ll do something about it.
We can’t seriously keep on accepting that « thousands of compromised devices » is a fine reality for a « small botnet ».
These devices should be quarantined.
Guess who has more votes, you or Amazon.
I don't even want to think about the ramifications for small and medium sized businesses. Realistically, how much would it cost to be able to completely destroy a local competitor by paying someone to orchestrate a few events in succession.
As I understand your argument, because the net has solid endpoints we can identify and isolate, we should ignore that fact. Instead we should create more and more complex systems to work around bad actors?
Bad actor takes control of grandma's computer. We should do all sorts of things except stop talking to grandma's computer? The thing, I would suspect, that most people would expect?
Businesses suffer from too much transparency. Got that part. They buy things that don't work and sometimes hurt people, even if they don't intend to do this. So far, so good. Where is the part where new businesses models are supposed to exist because some people made bad choices and the current models don't work? Why don't we just publicize the bad choices and let things work themselves out?
Sorry. Missing it.
My personal want / solution would simply be "everything gets an IPV6, and IPV4 gets deprecated. Everything using IPV4 gets an algorithm slapped on top to covert it into IPV6.
Dynamic ips become a thing of the past.
But I realize that is significantly easier said than done. (Makes Minecraft servers easier to setup though)
There is no need for any central authority and no need to maintain any lists.
Is there a reason other than inertia for why it hasn’t been implemented?
And the other ISPs like getting paid for DDoS mitigation, so they also look the other way. There's no money to be made fixing the underlying problem.
If you store 1 bit (banned/unbanned) + a unix timestamp (ban expiration) for each of those IPs, that requires more storage space than exists many billion times over.
To store such a block table you propose would require more memory for routers than any router has ever had and ever will have.
An attacker could easily "flush" all entries in this table by, for example, banning a TB of ipv6 addresses from talking to them, surely resulting in all participating routers dropping other bans to store some of those.
For example, if an attack is coming from a country you where you don't have many paying customers, but where there are many infected devices due to use of pirated outdated software, it is easier to ban the whole country than to figure out who is infected and who is not.
We can set a limit of ban records per host to prevent it.
Most of the time I see attacks lasting 15-20 minutes. I'm assuming it's either someone doing it "for the lulz" or some cyber warfare outfit testing their big guns.
I always consider the possibility of someone using DDOS to mask a more sophisticated attack.
1) Egress filtering by the ISPs
2) Better malware resistance and vulnerability mitigation on easily-compromised appliance and IoT devices
But neither is going to happen. 1 is a coordination problem. It has to be all or nothing, which can only be compelled by law, and we have no global laws and no global law enforcement mechanism. Some countries inevitably don't care and the rest won't partition the entire Internet by permanently cutting them off. 2 would probably make the entire Internet of Things and a whole lot of home computing just not happen because it isn't economically feasible. Poor security effectively acts as a tacit tax. We all pay a little bit each, but the tax is collected by criminals instead of governments.
Note that even your proposed solution here only works if 1 happened. Otherwise, source IP spoofing easily defeats a blocklist.
You're not seeing any SYN flood, just a bunch of TCP connections (equivalent of say search crawler), that are encrypted. Only after unpacking on loadbalancer those are visible as one TCP stream sheltering thousand HTTP2 streams.
Let's go back to username and password. 2FA forces scammers to up their game.
What about password managers? Having separate passwords to every account makes hacking into your accounts much harder and might hurt everyone in the long run.
And don't get me started on end to end encryption. Privacy, long term, will mean the fall of civilization.
Sarcasm aside. I think I understand your point in which we shouldn't just delegate to cloud providers the whole effort in preventing attacks, but just with everything production-grade, the average enterprise just isn't ready to deal with all the upfront cost to run your entire computing solution. Because it doesn't end with this type of mitigation and dependency. A similar argument could be made for not using proprietary chip designs made by cloud providers. Or any proprietary API solution for that matter. It really is a matter of convenience that a community solution might cover in the future, abstracting away fundamental building blocks every cloud provider must have (name resolution, network, storage and computing services) to provide such higher level functions without lock in. We are just not there yet.
But ISPs do not want to adopt such protocol.
Say some IoT device that half of households own gets compromised and turned into a giant botnet. The news gets out and everyone throws away that device. Now they are still blocked over a threat that doesn't exist anymore... doesn't seem like a good situation for anyone.
I'd imagine that the website owners that want the attack stopped will soon want to figure out how to get traffic back since they need users to pay the bills.
Whats to stop someone from just making an app that participates in an attack when connected to public(ish) wifi networks and participating in attacks long enough to get those all shut off from major sites?
How does this stop entire ISPs from getting shut off when the attackers have managed to cycle through all the IP pools used for natting connections? (e.g. the Comcasts of the world that use cg-nat to multiplex very large numbers of people to very small numbers of IPs)?
We can add an "accept" packet that lifts the ban.
Also, how do you remove yourself from blacklist when banned by Google or Cloudflare? I guess here you use the same method.
> Say some IoT device that half of households own gets compromised and turned into a giant botnet. The news gets out and everyone throws away that device. Now they are still blocked over a threat that doesn't exist anymore... doesn't seem like a good situation for anyone.
Not my problem. Should have thought twice before buying a vulnerable device and helping criminals. As a solution they can buy a new IP address from their ISP.
But it costs space in the routing tables and that means replacing routers earlier. It's no wonder, especially if you multiply it by thousand customers.
"block all traffic from outside from this IP" is significantly easier than "block all traffic from outside from this IP to this client". And you need to do it per ISP client, else it is ripe for abuse.
And don't forget a lot of the traffic will come from "cloud" itself.
But you should own an AS for that?
> But it costs space in the routing tables
Not implementing my proposal leaves critical infrastructure unprotected from foreign attacks. Make larger routing tables. Also, instead of blocking single IPs one can block /8 or /16 subnets.
Brilliant! Why didn’t we think of that?!? MOARE TCAMS!!!
Also, in Russia for example, there is DPI inspection and recording of all Internet traffic and if it is possible in Russia, then West can probably do 10x more. Simply adding a blacklist on routers seems like an easy task compared to DPI inspection.
That also means that every router now has to maintain a connection table to keep track of all of the pending confirmations, and to periodically check that table for expirations so it can clean it up. Maybe not that bad for a local router, but this is completely unworkable for routers handling larger parts of the internet.
And of course, anyone who has a tap into that level can trivially spoof all of the correct replies so it's still not a secure mechanism.
Also, on a well-behaved networks that do not allow spoofing IP addresses, this check can be omitted.
Ideally with the token packet being larger than the initial packet, so it can easily be abused for a reflection attack... ;-)
> Also, on a well-behaved networks that do not allow spoofing IP addresses, this check can be omitted.
This is already not true for most networks, and in your case would've to be true for all intermediate networks which is just impossible.
In another post you suggest this should also allow blocking entire networks; how do you prevent abuse of that?
Your suggestion is anything but well-thought, it's a pipe dream for a perfect world, but if we'd live in one, we wouldn't have ddos attacks in the first place.
I do think your idea has merit though. But it’s still a long way from being a well thought-out solution.
They can then ignore it until their server melts (which takes care of the problem) or take honorable action if one of their customers is compromised. The S stands for service after all.
I can confirm this. I see web pages talking about redirecting traffic to scrubbing centers.
So, if my neighbour is infected and one of his devices is part of a botnet, I get blocked as well?
Null routing is available in some situations, but of course it's not very specific: hey upstreams (and maybe their upstreams), drop all packets to my specific IP. My understanding is null routing is often done via BGP, so all the things (nice and not) that come with that.
Asking for deeper packet inspection than looking at the destination is asking for router ASICs to change their programing; it's unlikely to happen. Anyway, the distributed nature of DDoS means you'd need hundreds of thousands of rules, and nobody will be willing to add that.
Null routing is effective, but of course it takes you IP offline. Often real traffic can be encouraged to move faster than attack traffic. Otherwise, the only solution is to have more input bandwidth than the attack and suck it up. Content networks are in a great position here, because they deliver a lot of traffic over symetric connections, they have a lot of spare inbound capacity.
No. Your email will go straight into trash because ISP is not interested in doing something for people who don't pay them money. Also, even if they cooperate, it will take too much time.
> Null routing is available
Null routing means complying with criminals' demand (they want the site to become inaccessible).
> it's unlikely to happen
It will very likely happen if there will be a serious attack on Western infrastructure: for example, if there will be no electricity in a large city for several days, of if hospitals across the country won't work or something like this. Then the measures will be taken. Of course, while the victims are small non-critical businesses, nobody will care.
> Otherwise, the only solution is to have more input bandwidth than the attack and suck it up. Content networks are in a great position here, because they deliver a lot of traffic over symetric connections, they have a lot of spare inbound capacity.
So until my proposal is implemented the only solution is to pay protection money to unnecessary middlemen like Cloudflare.
I'm curious about your rationalization for this. Lack of privacy will also mean the fall of civilization. Civilization is just doomed to fail at one point or another. All things come to an end.
Yes! All things come to an end and that is why some recent philosophers think that Plato was naive to think it could minimize or erradicate society rotting. This is where negative utilitarianism comes in, where the point of society is not to maximize happiness (and therefore prevent society from collapsing) but to minimize suffering (and therefore provide mechanisms to minimize damages from transitions between organization forms when society collapses). I have to refer you to Karl Popper's The Open Society for this, because needless to say this answer is very reductionist.
"Those who would give up essential liberty, to purchase a little temporary safety, deserve neither liberty nor safety."
2FA and password managers didn't make us heavily reliant on massive companies.
Retool: https://arstechnica.com/security/2023/09/how-google-authenti...
If Lastpass goes away, people will still be able to use keepass or any of the large number of open source password managers, some of them even with browser integrations.
If I have a website that is frequently attacked by botnets and Cloudflare goes away, what can I use to replace it?
https://news.ycombinator.com/item?id=31652650
My response was to illustrate how insidious big companies are.
Of course nothing compares to the backbone of the web going down. If AWS North Virginia suffers widespread downtime to all its availability zones, much of the web will just go dark, no question about it.
But Lastpass doesn’t represent the whole of password managers. Storing your passwords in an online service is a really silly thing to do (for passwords that matter at least). Use something local like keepass.
Not using cloud is just very expensive and time consuming for the average user.
What I think would make this approach hard is that you would have to ponder if a newly created account is important at creation time in order to know if you should update the off-site, physical copy of your most important passwords (I say this because if you want to backup everything and avoid the cloud entirely it is just not viable, having to update this physical backup for each new account. I am currently at over 400 logins in my pw manager, 2 years ago it was half as much).
I think having your passwords encrypted with a high enough entropy master password and a quantum-resistant encryption algorithm, and having an off-site, physical backup of your cloud account credentials is enough for anyone not publicly exposed, like a politician or someone extremely wealthy, even though I would be skeptical these people go through such lengths to protect their online accounts.
So yes, I feel comfortable with my strategy of having backups on bluray disks + S3. If AWS goes down or decides to jack up their prices to something unacceptable, I will take the physical copies and move then to the dozen others S3-compatible alternatives. I am not dependent on AWS.
But I am not interested in using Google Authenticator or Lastpass because that would mean that I am at their mercy.
* though OTP seeds don’t print, and you can’t export/print attachments. I don’t recommend LastPass for these and many other reasons.
- AWS
- Cloudflare
- Azure
- GCP
- Great Firewall of China
Maybe there was some truth about "the world market for maybe five computers", after all...
Back then, you got a piece of land, and really could do what you wanted with it. Build a business, farm, etc. some government taxes but nothing crazy. But you had to deal with criminals, lack of access to medical care, and lack of education.
Now to do the same, you have a slew of building codes, regulations, zoning laws, and are basically forced to have municipal services. Higher Taxes to pay the roads, police force, fire fighters, education services etc.
However, home owners can still just have an egg or vegetable stand at the end of their driveway. It won’t be the same as having a storefront in town, but it’s still doable without the overhead.
Similarly, as the internet matures, we’re going to see more and more overhead to sustain a “basic” business.
But you can still have a personal blog ran in your closet, for lower-level traffic.
The analogy isn’t perfect, but unfortunately as threat-actor’s budgets increase, so too do their quality/sophistication of their attacks. If it was cheap to defend against some of the more costly attacks, they would find a different vector.
The answer, to me, is some tangential technology that is some mix of federated or decentralization. Not in a crypto bro sense, but just some tech whose fundamental design solves the inherit problem with how our web is built today.
Then threat actors will find another way, rinse and repeat…
No you can't. That is illegal without a "cottage food" license, training, and labeling in most of the US.
Garage sales often have a specific carve out, also, and limitations on numbers of time per year, etc.
Most areas nobody cares at all until it becomes a nuisance somehow.
Because selectively enforced laws are just another way of saying you have a king at some level, the person who decides to enforce or not.
However, the Supreme Court has left the prescribed remedy intentionally vague since 1996, which in turn makes the claims themselves less likely to be raised, and less likely to succeed.
https://wlr.law.wisc.edu/wp-content/uploads/sites/1263/2022/...
i'm not saying i like having to put the majority behind the services of 2 or 3 companies, but if you ever get shut down from some DDOS, you'll understand why people think they need to.
It does take enterprise grade tools to defend against the largest DDoS ever attempted.
Those are not the same thing. And those DDoS’s often are aimed at things besides a HTTPS endpoint.
Let's do it. It works for the website you're using right now. 2FA was in large part motivated by limiting bot accounts and getting customers phone number.
I can't imagine how much productivity the economy loses every day due to 2FA.
>I can't imagine how much productivity the economy loses every day due to 2FA.
Is it really that much? Every few days I have to enter a 6 digit number I generate on a device I have with me all the time. Writing this comment took me as much time as using 2fa for a handful of services for a month.
I use more than one service a day, and some infrequently, so for me about every day I have a minute or two where I try to login, need to find my phone (it's not predictable when it will ask), and then type it in. This happens to every person several times a day!
I also now must carry a smart phone with me to participate in society.
But the main drag is that when people lose or break their phones the response is: "just don't do that" and the consequences range from losing your account to calling customer service.
> Mostly TOTP. Bots could do that too. I don't see the connection.
Most people using 2FA do not use TOTP, they use a phone number.
Bots could use TOTP, it's more infrastructure, and it's a proof of work function for them to login.
You sign up for a Skype account or Twitter account and decline to give your phone number, instead choosing a different form of 2FA? In my experience your account will be blocked for 'suspicious activity' even if you have literally no activity.
It doesn't, you can regularly see people getting their accounts stolen here. This wouldn't be possible (or at least this trivial) with any competent implementation of 2fa.
That’s not a fair point.
We’re not even trying to make the internet safe. There is zero (0) actions being taken to stop this madness. If you run a large website, you still regularly see attacks from routers compromised 3, 4, 5 years ago. Or how a mere few days of poking around smartly is still enough to this day to find enough open DNS resolvers to launch >500Gbps attacks with one or two computers.
Why are these threats allowed to still exist?
The only ones attempting something are governments shutting down booters (DDoS-as-a-service platforms). But that’s treating symptoms, not causes.
We will eventually need to do something, or it will be impossible to run a website that can’t be kicked down for free by the next bored skid.
Just like paying protection fees to the mafia was a status quo, this also is just that. A status quo, not an inevitability.
The solution is to finally hold accountable attack origins (ISPs, mostly), so that monitoring their egress becomes something they have an incentive to do.
> The solution is to finally hold accountable attack origins (ISPs, mostly), so that monitoring their egress becomes something they have an incentive to do.
Be careful what you wish for. The sort of centralized C&C infrastructure and "list of bad actors everybody has to de-peer" that you would need to this effectively would we a wonderful juicy target for governments to go, "hey, add [this site we don't like] to the list, or go to prison".
Aren't there already a dozen or so such lists? I don't see how one more list really increases the risk.
You can make the list public - most of the bad actors are obsolete, compromised equipment for which the owner is unaware of the problem. Once the list is public, it's pretty easy to detect anyone trying to abuse the list as a tool of censorship.
And I do count that in.
Just because a user is the source of an attack unknowingly doesn’t make it right.
What would make it right is for there to be a more generalized remote blackholing system in place.
ie my site runs on an IP, is able to tell my ISP to reject traffic to it from $sources, and my ISP can send that request to the source ISP.
And if it makes my site unavailable to that other ISP because of CGNAT and 0 oversight, tough luck. Guess their support is getting calls so maybe they start monitoring obviously abusive egress spikes per-destination.
Yes, you're 100% correct. Back in the day when the main bot net activity was spam if you were infected and you started sending TB of spam the ISP would first block your outgoing smtp. If they kept getting complaints in a week or two they'd cut you off.
I remember 30 years ago when most people were on dialup, I was fortunate enough to have 128kB SDSL. As a relatively clueless kid I decided to portscan an IP range belonging to a mobile service company. Few days later my dad got a phone call saying their IDS flagged it and "don't do it or we'll cancel your service". For a port scan of few public IPs no less!
ISPs could definitely put a stop to 99% of these botnets, but until they see some ROI, why would they bother?
There’s no practical action being taken besides « use our profucts cause we can tank it for you » here.
The mitigations listed are better than nothing, but the fact that every skid out there can hire a botnet of a few thousands compromised machines (like here) and send you a few millions (say this protocol attack allowed a 100x higer than avg impact) rps is way enough to kill the infra of 99.99% websites. No questions asked.
You can see those paradoxes at play throughout the corporate world and especially when it comes to actual combat/war (to which actual combat/war these DOSes might actually be connected). For example the fact that Israel was relatively successful in implementing its Iron Dome shield only incentivised their adversaries to get hold of even more rockets, so that the sheer number of rockets alone would be able to overwhelm said Iron Dome. That's how Hamas got to firing ~4,000 rockets in one single day recently, that number was out of their league several years ago when Iron Dome was not yet functional.
Procuring and operating the infrastructure to mitigate this kind of attack costs many many thousands of dollars or requires becoming part of the Cloudflare/AWS/Google hive.
Joe Schmo can set up a TOTP server, run keepass/bitwarden and use letsencrypt for free (or another SSL provider for cheap).
The lament from parent is that running a simple blog reliably shouldn't require being inside Cloudflare's castle walls or building your own castle.
---
My personal observation is that simple websites should continue operating HTTP1!
So in a way this was partially caused by the existence of insanely big tech companies that need such features.
Well, I wrote this comment to ridicule yours... but actually that was what happened.
DDoS is really the only thing that you can't host yourself on your own machines in today's internet.
It's the spam that killed email, not the filters.
What happens when you log an attack from a device that is attacking you from a school or business WiFi network? Block the whole IP forever?
What if the user is on a CGNAT. Are you going to block the edge proxy for that entire ISP?
What if you're getting hit from a residential connection that gets a new rotated IP every couple of weeks? Block whoever gets that IP from now on?
Your solution doesn't stop attacks. It just stops regular users.
No, but for a day perhaps.
> What if the user is on a CGNAT. Are you going to block the edge proxy for that entire ISP?
Maybe. If the ISP doesn’t bother doing anything about it (which is THEIR job, not mine as a website operator).
If the ISP can’t be arsed to do their job, why am I supposed to care about them at all?
> What if you're getting hit from a residential connection that gets a new rotated IP every couple of weeks? Block whoever gets that IP from now on?
Same as the CGNAT one. It’s the ISP’s job to handle their misbehaving customers.
If they refuse to do it and get complaints from their other customers that they’re getting blocked, maybe they’ll actually get to it.
> Your solution doesn't stop attacks. It just stops regular users.
No. It puts pressure on the ISPs to finally stop whining loudly when they receive an attack while closing their eyes on any attack originating from their network.
This is not sustainable.
And maybe Facebook and Google are big enough to push around the ISP's, but they are the only ones. Nobody will bat an eyelash if 15,000 Comcast users in Phoenix AZ can access your hokey-pokey website. Comcast doesn't care. The users won't blame their ISP. They will blame you, or whoever owns the hokey-pokey website. If you want traffic, you need to be equipped to handle traffic. You are the one with the internet facing infrastructure.
You are the one blocking traffic. Not the ISP. That is how it should be. The ISP should be impartial. You pay for connectivity. Consider yourself connected. For better or for worse. You are responsible for what you put onto that connection.
They do already. DPI on port 53 for DNS blocks or SNI inspection are common place. So are IP blocks.
> If you want traffic, you need to be equipped to handle traffic. You are the one with the internet facing infrastructure.
Slightly misleading wording here. More accurately your point is: « you want to run a website? Better have the infra to support traffic spikes comparable to that of a tech giant ». 400M rps would cost an unfathomable amount of money to be able to handle even just while dropping all packets.
> And maybe Facebook and Google are big enough to push around the ISP's, but they are the only ones. Nobody will bat an eyelash if 15,000 Comcast users in Phoenix AZ can access your hokey-pokey website.
Obviously yes. Too bad it’s better business for everyone to say nothing and just recommend you use their product.
Let Google, Amazon, and Apple decide who gets to use the internet and who gets put into a list.
That is way worse than giving Google the W3C. That is literally just handing them the internet and making everybody else on it subservient to Google.
It would be nice if the cell phone provider could send a text message reporting the problem. But how to distinguish it from spam?
The fact that millions, if no more, devices can continue to access the internet regardless of how long they are compromised, is just crazy. I get that it put more responsibility upon end users to secure their devices, if they otherwise run the risk of get thrown of the internet, but I currently fail to see other options. Our device security still isn't good enough that we can just use them with reckless abandonment.
Any "solution" that attempts to fix the problem of increasing DDoS attacks and their damage that doesn't address the issue of compromised devices being allowed to roam free on the internet is a band aid at best.
And I can almost hear people complain that I'm arguing to throw compromised IoT, SCADA and monitoring devices of the internet, and yes I am. None of these things have any business being exposed to the public internet anyway.
Currently there are zero rules (outside of a ISP ToS maybe) that forbids what you’re talking about. Pretty much anywhere I think? Unless you know of a law against having a infected or out of date computer connected to the internet?
There really is no way to have both. The current situation, they generally only deal with problem cases that get reported to them. And I doubt anyone is going to bother doing so for the 20k machines in this attack.
> No, but for a day perhaps.
Then that's also a DDoS attack vector.
But, a slight defense of it—the really big providers can already sink a massive DDoS anyway. So, this is just a scheme to help little websites. It doesn’t really matter if a school, or even a cellphone network, can’t access my little website for an afternoon.
You’d have to decide if you want to send the block request. If you are hosting your personal blog, you’ll probably go for it regardless. If you are providing a small service; hosting git for a couple friends or whatever, you’ll probably block with some discretion.
$NET gives away DDOS protection for free for non-businesses
If you want to complain about an actual working solution, that's your right, but realize that without an alternate solution you're advocating for giving small gangs the ability to disrupt everyone else's lives on a whim.
This is not coming from "known botnet IPs", this is from random infected devices. Some aren't even permanently doing this, just one request from a device per day - it already large enough to cause issues.
Improving the ability to track down and prosecute perpetrators tends to result in less anonymity/privacy, so that makes the problem challenging.
Thinking in the long/very-long term, we need to get more innovative with the underlying technology to mitigate abuse. I mentioned this effort https://named-data.net in another part of the thread.
Using any kind of community coordinated IP ban is useless and would hurt a lot of people, millions(or even billions) of devices have dynamic IP addresses.
You would not stop botnets from DDoSing you and on top of that you'd block millions of legitimate users.
Don't agree.
> the only solutions available are to pay Google, Amazon or Cloudflare a protection tax.
It's not.
> come through some community coordinated list of botnet infected IPs
How would that help?
The problem is that IP addresses are not a reliable identifier, especially for the kinds of folks whose routers have been infected by malware. Few ISPs hand out static IP addresses anymore. It's why online games no longer bother with IP bans anymore, because as soon as the target reboots their router they evade your ban and some other poor sap on the same ISP gets stuck with the flagged IP.
Vaccines inevitably lead to stronger viruses, but would you argue we should go back and not have began to use them?
Cloudflare and Google may be some sites' only hope to staying alive in the event of network-driven attacks. I suppose this landscape is a double-edged sword.
I imagine distributing the load becomes harder/more expensive the bigger the scale. In real network you probably have to find the paths that don't slow you down and then strike for short time before you are banned.
Cloudflare has made post about mitigating 2 Tbps DDOS in 2021, so in real attacks request sizes aren't quite that optimal.
"The good point in this is that this will probably make more people want to reconsider H3/QUIC [HTTP/3] if they don't trust their products anymore :-)"
https://www.mail-archive.com/haproxy@formilux.org/msg44136.h...
Normally for DDoS migitation, you blackhole on your normal ISP, and you simultaniously advertise on the mitigator. The mitigator scrubs the traffic and sends the clean stuff over a private session back to you.
If you just black hole and move on, yes that's a lose. However many ISPs will, because the quick reaction holds the most value for them.
That seems high/impressive
I guess we don't have a good understanding if somebody had control of a blend of 10k VMs in the cloud, 10k infected PCs, etc.
We are an old server. You have to be nice to us before we do more for you. -.-
How did Google mitigate it exactly? Is there a technical deep dive?
> To mitigate against the non-cancelling variant of this attack, we recommend that HTTP/2 servers should close connections that exceed the concurrent stream limit. This can be either immediately or after some small number of repeat offenses.
https://cloud.google.com/blog/products/identity-security/how...
Most web services don't need clever protocol attacks to be downed by 20k machines.
Mitigations are just that, mitigations. They are as effective as buying a better door lock to protect your apartment from a nuke.
What would be truly terrifying is if Google kept this a secret and let it sneak up on the world. I have zero faith in Google to do the right thing, and I have zero confidence in their impartiality, and zero confidence in them being the gatekeeper for internet standards. And that's WITH all these silly open self-congratulating blog posts.
Google needs people to trust them in order for them to try to be gatekeeper of best practices and web standards. They want nothing more than to absorb the W3C. That will never happen if they can't convince the "professionals" to parrot everything they say. And they can't get anyone to parrot unless they write these self-congratulatory blog posts. Likewise, they need all the open-source developers who basically wrote Google's codebase to fix their code for them, for free of course. They won't do that unless Google tells them what's broken.
EDIT: Ok, so this was a 0-day issue. Then it all makes more sense. Sorry.
Can such an attack be performed with botnets?
Most likely the figure is incorrect, or at least misleading.
Cloud Flare also got one of these, 201 RPS.