That’s strange, are you actually compiling packages or the evaluation itself ooms?
Because if the former, you may help it by limiting concurrent builds.
Memory is eaten by lazy evaluation of NixOS modules.
It can be optimized, but it is not a low-hanging fruit. It is still waiting for its hero. I'd compare it with creating the V8 for JavaScript.