It might indeed be the RAM. I've never seen this behaviour on my M1 Max with 64GB.
Faster drives help a little, but the most practical option is to have enough RAM to use no swap at all.
I have a 64GB Mac and I'm regularly over 32GB. If I only had 32GB my ten core Mac would still grind to a halt.