Disabling TCP offload to hardware makes this comparison much less useful. TCP has hardware offload right now, and QUIC will take years to get at that level.
RFS flow tables are installed by sendmsg and recvmsg, so I fail to see why RFS doesn't benefit UDP just as much as TCP. If you are using RFS, your QUIC packets should still arrive on the correct CPU.
Doesn't that mean that you should have more NICs?
From my recollection, the main thing that's made a computer a server or mainframe for the last 50 years has been that such systems have enough IO channels to saturate the CPU (i.e. they're intended to be CPU-bound, not IO-bound, under heavily-concurrent workloads.) If you take all the IO-offload cards out of a mainframe, it's just a regular computer.
For example, this Cascade Lake Xeon has 48 lanes of PCIe gen 3: https://ark.intel.com/content/www/us/en/ark/compare.html?pro...
If you give yourself 32 of 48 lanes for networking, that's only 256 gbits of PCIe bandwidth. So each 16x slot is limited to a 100 gbit card.
And that's a 22 core processor. With DPDK, coming up with a application that could process 200 gbit is almost trivial.
While the talk is centered around RISC-V, there's nothing RISC-V specific about this, just that it enables whimpy cores to control multi-Gbps traffic.
[1]: https://www.youtube.com/watch?v=LDOlqgUZtHE (Accelerating Computational Storage Over NVMe with RISC V)
Sure, there's the rare case of a connection moving to a different L3/4 flow identifier causing a cache miss, but that's not something that happens often enough to be a relevant bottleneck.
The amount of TCP hardware offload in linux right now is actually fairly limited. Kernel developers have largely rejected true in-hardware TCP stacks for a while now (although you can get this functionality with high end NICs and custom kernel builds). That said, since QUIC generally reduces the ack frequency it should be possible to match or exceed TCP with hardware offload. Especially when you look at total throughput with many streams open.
QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.
1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
I'm trying to find the actual general benefit, but so far I'm not finding it. There is a technical benefit, but not on the scale of say h2 multiplexing or binary vs. plain-text protocols.
I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.
Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take.
Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.
https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc...
Generally, when we saw losses, we were able to work with the datacenter to track down bad cables, or overloaded paths, or overloaded fabrics (which was usually in their 'legacy' datacenter). I expect the datacenter networking was overprovisioned (separate backend network, dual NICs on hosts, multiple uplinks on switches, etc), but it was darn reliable. Of course, it would have been nice if the people who owned the datacenter would have been monitoring all the stuff their customer (us) was able to find was broken. And, some sort of multistream connectivity would have helped avoid the paths with loss; ocassionaly the loss would be bad enough to impact service, and we'd need to force reconnects until the tcp stream went on a different path.
By saying it's "inappropriate for datacenters," I think he painted too broad a brush - obviously not all are the same, and not all run at Google scale. Depends on the datacenter and the workload.
TCP is baked into firmware on edge routers and firewalls and wifi devices that are beyond the period of support and the companies that created them aren't interested in supporting them or have gone out of business.
There are other protocols in layer 4 that could have been used that are better on some workloads than TCP, but TCP and UDP were the only things configured/supported; many firewall default configurations filter out things they don't recognize so anything new is a non-starter. And because of the perception of responsibility when it comes to networks, if I support a new protocol and you can't connect to it because there's older devices between you and me, it becomes my problem.
Anyway, anothing not 'normal' is being filtered out somewhere. We even (at one startup) considered tunneling RTP through HTTP! My god.
> One concern about the move from TCP to UDP is that TCP is widely adopted and many of the "middle-boxes" in the internet infrastructure are tuned for TCP and rate-limit or even block UDP. Google carried out a number of exploratory experiments to characterize this and found that only a small number of connections were blocked in this manner.[3] This led to the use of a rapid fallback-to-TCP system; Chromium's network stack opens both a QUIC and traditional TCP connection at the same time, which allows it to fallback with zero latency.[17]
Of course, you could just buy a TLS accelerator if you wanted one.
Another way is if it takes a couple CPUs to run my application, a CPU or two to drive the storage, and a CPU to do TLS, why would I also spend a CPU on TCP if my network card already has one that does that?
That's a little bit like what programmers are talking about when they are talking about "multi-core programming", except those are general-purpose "cores", and (presumably) these "cores" can only handle doing TCP.
I'm not sure about the implementation details of GSO and TSO in Linux. But I know a thing or two about the hardware: the Intel Niantic/82599 NICs support UDP transmit segmentation offloading, I've checked the datasheet real quick and it says "Note that current UDP segmentation offload is not supported by any standard OS."
Not sure if this is just an outdated comment in a 10 year old datasheet or if there's some performance to be gained by using this UDP segmentation offload for quick.
Edit: Should have clicked on the link in the article [1] before posting here, UDP segementation offloading is coming to lots of NIC drivers in 5.5.
Always interesting how the hardware often supports way more features than the drivers expose and you suddenly get new hardware features on a 10+ year old NIC model :) (Similar story with IPsec offloading on the Niantic/82599)
It's also supported on the old chipset from the motherboard I bought more than half a decade ago, and every chipset I've used in the server space in as long as I can remember.