rigtorp

With the right cooling setup I've been able to get Xeons to run permanently in turbo mode, kind of a back door overclock. You would have to experiment.

For lowest latency applications I void avoid using RT priorities. Better to run each core 100% with busy waiting and if you do so with RT prio you can prevent the kernel from running tasks such as vmstat leading to lockup issues. Out of the box there is currently no way to 100% isolate cores in Linux. There is some ongoing work on that: https://lwn.net/Articles/816298/

Oh yeah, the whole NUMA page migration stuff is not well documented. You'll also proactive compaction to deal with in the future: https://nitingupta.dev/post/proactive-compaction/

That would work with a heat pump setup. Liquid nitrogen cooling is usually done by evaporating into the air directly on the processor package (as far as I know). So you would always have condensation inside the server case. Hmm, I guess a isolated duct for evaporated nitrogen to escape and heating the outside of that duct to ambient temp.

Disabling swap only prevents major page faults on anonymous memory. You want to avoid all page faults by using mlockall (https://linux.die.net/man/2/mlockall). At that point swap settings doesn't matter. But yeah, disable swap anyway just in case.

SMT sibling threads can definitely impact each other. It works great for common workloads. If you have a highly tuned workload with high IPC or want to trade off throughput for latency, disabling SMT can be a win. Disabling SMT also increases effective L1 and L2 cache which can be beneficial.

HT share most execution units in the core. If your workload stalls a lot due to branch misprediction or memory access (low IPC) these units can be shared effectively. The Linux perf tool can be used to check IPC.

You might be interested in https://rigtorp.se/virtual-memory/ where I look deeper at the VM subsystem.

I don't consider that file backed since they pull memory from the same pool as anonymous memory and not the page cache. The Linux kernel docs makes the distinction between file backed and anonymous memory. I think a better term would probably be page cached backed memory vs anonymous memory.

Yes and also subambient using heat pumps, but I have never seen it deployed in a data center. How would you deal with condensation?

I've always been using kernel bypass for low latency networking. You can also use SO_BUSY_POLL with the Linux stack. I should at least mention this in the guide.

With busy polling you basically halve the SMT sibling thread's memory bandwidth. But yeah it might work well for a specific usecase anyway.

Most of these tips apply to all architectures. The only x86 specific parts are regarding CPU power management and turboboost.

RHEL, but all the tools are open source.

Many organisations run water cooled overclocked servers in production. I have not yet heard of any production use of sub-ambient cooling, but that would be awesome!

I have another article on virtual memory: https://rigtorp.se/virtual-memory/

For lowest latency applications I void avoid using RT priorities. Better to run each core 100% with busy waiting and if you do so with RT prio you can prevent the kernel from running tasks such as vmstat leading to lockup issues. Out of the box there is currently no way to 100% isolate cores in Linux. There is some ongoing work on that: https://lwn.net/Articles/816298/

This guide pretty much tells you how to make the Linux kernel interfere as little as possible with your application. How to instrument and what to measure would depend on the application.

I agree that measuring queuing delay and processing delay separately makes sense.

For a truly lowest latency in software application you need to avoid all context switches. Using interrupt driven IO adds to much overhead. You need to use polling and busy waiting. I'm working on a guide for this type of application design. If you are indeed using the Linux network stack, then yes adjusting NIC interrupt coalescing and interrupt affinity is useful.

Yes, definitely turn off HT/SMT and use a single app thread per core with busy waiting. I'm working on a low latency application design guide exploring this more in depth.

Yeah, split locks have a huge performance impact, especially with high core counts: https://rigtorp.se/split-locks/

Fortunately the Linux kernel can kill offenders with SIGBUS: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Sp...