I tried setting up nix-darwin with home-manager, but it just made the system more confusing to operate. Ended up using home-manager standalone without flakes. Works like a charm.
I run everything through an idempotent 'rebuild' script, so I don't really notice any day-to-day difference between nixos, standalone HM, or nix-darwin.