Hacker News Reader

Comment by blibble | Hacker News Reader

indeed

I suspect a single machine could put out 71M HTTP requests/second without too much difficulty

(but cloudflare never misses an opportunity for a blog post)

71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget.

For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).

> So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget.

fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue

generating requests is a lot less CPU intensive than parsing requests

how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all

(the TCP negotiation will likely be the bottleneck)

I've interpreted the "put out" in the initial comment as in "put out a fire" (i.e. mitigate), instead of "send out".

Could you do this and write it up? I'd love to see how you tune this stuff in hardware and software to get that sort of throughput.

These were HTTP/2 requests, not connections, so no per-request TCP negotiation. I'd say it's very easy to generate a huge number of requests like this, even if it's over TLS.

But it's not one machine if you do anycast.

It's a lot, but at cloudflare's scale you either have the budget for a crapton of machines per point of presence (hope I'm using the term correctly) or custom hardware that can deal with this sort of thing. It's kind of their core business.

I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either.

Edit: I misunderstood the comment and thought about "ingestion" and not "generation"

You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps.

That is 35 10G nics at wire rate.

>I suspect a single machine could put out 71M HTTP requests/second without too much difficulty

PacketsPerSecond=TransferRate(bytes/second)/PacketSize(bytes/packet)

https://community.arubanetworks.com/community-home/digestvie...

A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets.

If you are in the business of being the main auditor of your activities, water companies here in the UK being an example, news organisations as another, then does one ever find a problem with one's work?

I've had DDOS on my systems before, and how it took my firewall out was simply increasing the number of requests the firewall had to handle using a SYN flood. https://en.wikipedia.org/wiki/SYN_flood

Most firewalls are fixed and not adaptive, so a bit of low hanging fruit could be had by firewall systems that can adapt to SYN floods, like shortening the time the handshake process stays in memory, choosing to dump all that dont complete within a short period of time, choosing to dump those from ip address blocks linked to country's or regions your dont normally trade with. There are so many ways to handle these situations, but not all firewalls and switches provide the facilities that are needed.

Hackers will exploit default configurations and industry standard best practices, including RFC's.

During one of my lets see what happens if I push this metaphoric button exercises, I was tracing those random packets that appear at firewalls and I happened to have picked one which tracerouted back to a French IP.

What was interesting and consistent with behaviour I've seen in the 90's with dead submarine internet connections at the New York exchange, was the trace route delivered its hop data until I crossed the English channel at which point a packet killed the machine, forcing the mains to be pulled as the machine went totally unresponsive.

It makes me think the UK internet is like some giant virtual dome like the one seen in the film The Truman Show.

>"A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets."

This is not correct. An interface doesn't handle more packets per second with smaller packet sizes it handles less, much less. Getting the full 48 Gbps of 24 ports at line rate would only be possible if the packets were full size packets(1500 bytes). The PPS bottleneck with small packets(64 bytes) would mean you would never see line rate on those 24 interfaces as your throughput would fall off a cliff. Network hardware vendors quote PPS using the lower bound of packet size and line rate at full packet size. The PPS bottleneck on a network device is incurred long before the line rate limit is ever reached.

The min network packet size is 64 bytes, so if they are bundling several small packets into one pseudo jumbo packet and handling it that way, something only using the same manufacturer switch could do as this would be propriety networking, then that could explain their calculations on their webpage.

The data is streamed within time slots, which is why cable lengths of Cat<=5 cable can not be longer than 100 metres @ 100Hz. Cat 6 is 55metres at 250Hz.

I dont know what their switch internal workings are, you'll have to ask them why they have quoted that speed, but I know some switches also have built in fibre network connections, as invariably with a network switch with many ports, you could get a queue of packets for one port so there must be some sort of buffering inside a switch to handle and minimise these situations.

It is only "clockwork" electrical pulses at the end of the day.

This is all complete gibberish. Packets per second is based on the lower bound and Jumbo Frames denotes an MTU or upper bound. Further switches don't aggregate smaller Ethernet frames into Jumbo frames. Packet per second also has zero to do with an Ethernet cable's length. PPS is the "forwarding rate" of the switch which is distinct from the bandwidth of a switch's backplane or switching fabric. Again switching bandwidth and forwarding Bandwidth are two different things. There is also no need for me to "ask them why they have quoted that speed", that's standard with hardware vendors for PPS.

You won’t get more than tens of thousands of concurrent network connections at most per machine. Your network switch is likely to fail first, or your operating systems ability to manage concurrent requests. 71M is a lot more network than you suspect.

Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit.

But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.

Reply on news.ycombinator.com