Turn off hyper-threading and boom, everything started to function normally.
The solution to not having enough food for one mouth often isn't to add another mouth.
I still see this in modern benchmarks on modern systems where multicore throughput on many benchmarks is just better when you just turn off SMT.
AMD's 3D chips go faster because of the huge cache. Every time we add another process to a core we just split the cache in half instead.
Clearly, this is workload dependent. But I immediately turn off SMT on new systems I get. I'm not core limited, I'm usually single thread limited. Like, always. No reason to keep SMT on for this case.