> moving builds to my desktop to use a hardware signing key and uploading them in a non automated, non transparent fashion is an improvement on securityFor most projects it is an improvement, for better or worse.
First issue: private keys stored in files can be stolen silently, and then the only recourse is revocation. That's the main reason for the HSM requirement: malware authors have been doing this for some time now and revocation is difficult/expensive for various reasons. An HSM can also be stolen but only in the old fashioned way of breaking into your office or home and grabbing it, which you're going to notice.
You may object that the credentials for using the HSM can be stolen, and that's true, but they can also be changed easily and quickly. So if you notice that your PIN has been keylogged, you can recover from the compromise then change the PIN and you're done, no need to revoke the certificate.
Second issue: automated signing in CI can actually be risky. It means anyone who can push code to your CI system can get code signed as yourself, possibly without you even being aware of it. The key is held online at all times, so obviously if the CI system gets hacked then it's game over, but even without that it boils down to anyone who can push code into the system becoming a weak point, especially because CI systems are running lots of arbitrary code without being closely monitored. CI signing is at best 1-factor security.
If you sign locally then the key can be (literally) offline until the moment you do a release, and access to it can be constrained via 2-factor auth: the key is something you have, the credential is something you know. So this is quite secure.
For signing nightly dev builds, internal tools and other transient binaries that shouldn't get out into the wild anyway, you can self-sign which is free.