I don’t know about CircleCI but if you can persist the /nix/store then that will give you the performance improvements you’re looking for.
I use Gitlab CI and with self hosted runners I can make sure the /nix/store is persistent across CI jobs and then my builds take ~5 seconds if no code has changed, 5 seconds of which is just nix build evaluating all my nix files and realising no work needs to be done.