How to achieve low latency with 10Gbps Ethernet
blog.cloudflare.com
blog.cloudflare.com
Which is good, because often tuning means making trade-offs. A lot of those things like polling and interrupt shuffling have side effects and might for example negatively impact other scenarios -- presumably not everyone out there is running single client-server ping-pong with small UDP packets.
Yes a lot of parameters have retained default values from back in the day when system characteristics were different. Switching defaults drastically might break existing code that relied on them. Perhaps those are better updated during major distro version upgrades (RHEL 6->7 for ex).
In general though, I would say Linux is tuned for average throughput not latency. Sometimes they go hand in hand, sometimes they don't. Over the years there have been patches and forks related to operating on low latency environments (stock trading, media processing, hardware control, graphics in desktop environment). Those range from parameters tweaks in /proc to kernel patches, to almost full kernel bypass via things like DPDK for Ethernet for example.
So perhaps there is a use case for wizard like configuration tool for intermediate users. It can ask you questions like -- how many clients do you have? How many concurrent connections. Would you rather have higher average throughput (say for downloading files) or lower packet latency (say for audio processing). Would you rather use your memory for a file cache for for your application data. And so on.
Next step up -- a self tuning system. It monitors your usage pattern then decides what settings to use. Monitors again, and adjusts a bit and so on.
Regardless of whether you use DevOps or not, if scalability is an issue, someone needs to be doing performance testing and sizing to have a rough idea of how many users per node you can support. It's part of the QA cycle; and that doesn't go away with DevOps.
http://servicesblog.redhat.com/2012/04/16/tuning-your-system...
Great article regardless. I'm wondering what the latency would have been if they had started with that step.
Edit: other than needing the thousand-dollar NIC, I mean.
It's a great article, but I'm surprised that the author introduces spinning and task-setting without noting CPU isolation (either with cgroups or isolcpus). Not doing so could introduce problems.
But so much the win come from being userspace and tunable. OpenOnload also accelerate pipes, loopback UDP/TCP and epoll, all of which are unrelated to their hardware. Indeed, having unaccelerated fds in the same epoll set as accelerated fds kills performance because it needs to ask the kernel for the status of the unaccelerated fds.
That said, I think you need to have a SolarFlare card for OpenOnload to work at all, even if you just use it for local communication. I've not tried it though.
This requires the creation of a lot of code to reimplement the kernel functionality you need by a software engineer of atypical skill to ensure a robust and performant outcome. In other words, it is expensive and requires a skill set that is relatively rare to use well. However, if you have significant server CapEx and OpEx it can totally be worth it to reduce your required server footprint.
BTW, you can do this with many common Ethernet chipsets, including the ubiquitous Intel ones, but they are all a bit different to work with.
Create a receive queue for each core on the NUMA node, spin up parallel processes (1 for each core and pin to the core), then aggregate the data. Wire rate monitoring with a $700 NIC and a commodity PC. Pretty amazing considering what companies used to pay for such performance.
Put another way, identify which socket the PCIe bridge and device are attached to, then make sure the DMA buffers, and interrupt handler are pinned to that node.
Also, don't touch the buffers from the other socket. Strangely enough touching the DMA buffers from the other node just once was enough to cause a performance fall off.
edit: finished reading, its the last step :/ and ~halves best case they were able to achieve with tweaks.
For the extreme cases you really want to control the packet allocation for receive and make the buffers your app uses be also used for the network transmit so you get a real zero-copy.