I love nix as a concept, but in my experience it isn't practical on systems without a ton of memory. I regularly get build errors due to oom errors.
In "normal" circumstances you don't generally have to do any builds at all - (almost?) everything should come from the binary cache. To be doing all these rebuilds it sounds like you're customizing packages, which is already something other distributions don't tend to let you do at all.
Memory is eaten by lazy evaluation of NixOS modules.
It can be optimized, but it is not a low-hanging fruit. It is still waiting for its hero. I'd compare it with creating the V8 for JavaScript.