You can also use something like agenix or sops-nix to deploy the secrets encrypted to the machine in the system closure.
Also, SCPing the file over is bad for reproducibility.
For the topic at hand isn’t this always a problem with deployment systems? You need to have the secret somewhere after all. In my case I only ever use nix for personal systems, so feel totally justified just storing my ssh key as a secret in yadm.