Unless you're paying me a lot of money (and even then) I WILL NOT MAINTAIN AN HSM FOR YOUR SERVICE. PLEASE FUCK OFF.

If you cared about security you would let me authenticate with ssh key signatures. GitHub does this, if you can manage to talk to an HSM you can manage to talk to the openssh agent.