Linux 6.6 Enables Tracking Per-CPU Cgroup CPU Usage Stats
phoronix.com
phoronix.com
1. We can now adjust how much of each cpu a process can use; this pull. Before we could use affinity to enable or disable core access, but now we can also say, only use a little of this core. Leave most of this core for others. More flexibility in figuring out scheduling plans.
2. Now we can change the boost behavior per core. So we can set one or two or so super boosted cores & keep others a but lower. Then assign work to this high clock rate core or cores. https://news.ycombinator.com/item?id=37458227
3. Now processes will tend to stay next to their l3 cache, whereas before they wouldn't understand that they were moving across l3 cache zones. So now they'll tend to have hitter local caches. https://news.ycombinator.com/item?id=37458221
Major boost to the flexibility of workload handling in 6.6.
Ideally we'd see that branch figure out how to incorporate some of these ideas. I haven't done an analysis to explore. The article mentions one sample eBPF scheduler that does some cgroup work, but I haven't looked deeper.
This quote from the PR I think is a bit better for people with my smooth brain
> What are you adding?
I want to add a *per-cpu variable* to record the cumulative per-cpu time of cgroup and its descendants, which is similar to the variable "cgrp->bstat", but it is a per-cpu variable. It is very useful and convenient for calculating the usage of cgroup on each cpu, and its behavior is similar to the "cpuacct.usage*" interface of cgroup v1.
https://lore.kernel.org/lkml/3d2b68bf-9f40-c779-dcfd-4cf9939...
Id' still love to hear more about the use cases for this
Like Arm's big.LITTLE cpu designs, the last few Intel CPU design generations have used heterogenous CPU cores- some more powerful designed for fast execution, and other less powerful ones designed for power efficiency and density. AMD also has heterogenous cores now, too- with it's X3D chips utilizing chiplets with both stacked cache and without.
Unlike how it used to be on x86, where a reasonable approach was to just spread jobs out evenly across all cores, now which core things go to can have a huge impact on speed, with some games running better on the higher cache cores of an X3D chip while others do better with the higher frequencies of the normal ones.
In the future, I believe this will help reshape the backbone of thread scheduling to make it more efficient and have a better understanding of the processor it's working with without necessarily needing a significant amount of engineering work per heterogenous CPU that comes out.
Indeed. Ideally the kernel would intelligently pull processes that have lots of cache misses towards cores with more cache and processes with fewer misses towards cores with higher clocks. And do that with any finite resource that's unevenly distributed across cores, such as vector units and whatever performance counters can inform the scheduler about.
Sure, maybe heterogeneous isn't the right word, much like NUMA is often not heterogeneous. But these do make CPU placement matter.
If you have three software threads A, B, C and two CPUs X and Y, and thread A runs on CPU X from T0 to T2, thread B runs on Y from T0 to T1, and C runs on Y from T1 to T2, then thread A made more progress than the others, all else equal.
If all three threads are spending most of their time doing pointer chasing (dependent memory loads) then not really: each core has bandwidth to do more than one memory load at a time, so there's not resource contention and each SMT can make progress in turn on the shared core.
If all three threads are well pipelined math heavy computation, then yes, the core running two threads has to split its adders among the two threads and the core running a single thread doesn't.
If it's a mix of workloads, sometimes you get lucky and the two threads don't contend much and you get a nice throughput increase, and sometimes you get unlucky and they do contend but throughput is about the same. If you get really unlucky, the two threads running causes the working set (memory or instructions) to grow beyond the cache and you get a reduction in throughput.
But it really depends on your workloads.
This is also generally useful if you're using cpusets (even if you're not on a numa system) because you may have them configured in a way where some CPUs belong to multiple cpusets. You can already get per-cpu CPU usage information, but this could help you figure out why a particular CPU has more usage than another.