If your definition of an HSM is just "hides my key material from my application server", then sure. This is basically the idea behind HashiCorp's Vault.

However, you'd still be vulnerable to someone gaining root access on your Vault server. A true HSM uses physically secure memory and a dedicated cryptoprocessor to make it impossible for the secret to ever be leaked, even if the host OS is compromised. This isn't really something that's possible in software alone.

(In fact, the Vault documentation even mentions that you might still want to use a dedicated HSM for security, since they implement a plugable mechanism for secret storage.)

That said... you could use a TPM chip or ARM TrustZone to implement true TPM functionality. It's still a hardware based solution, but it's hardware that's much more affordable and possibly already inside your computer.

Not really, no. AIUI, the whole point of having a hardware security module is that it only leaves a very narrow attack surface (ie, the protocol for communicating with the device) for malicious software to try to exfiltrate the sensitive data through, with no way to get around it - so if that interface is sufficiently hardened, you should be secure regardless of any weaknesses elsewhere in the system.

In contrast, while you could think of, say, an ssh key agent as a sort of "software security module", such things are inherently vulnerable to low-level attacks that side-step the intended interface. Say, a kernel exploit that grabs the decrypted key straight from the agent's memory.

So if you need to keep your secrets safe even if you unwittingly use them on a comprised OS, you can only pull that off with a hardware module.

Yes, but then you lose the whole benefit of a HSM -- that keys generated in the HSM are physically protected from being extracted. Aside from that, a HSM is just an implementation of regular crypto algorithms.

Yes you can and I know at least two companies that do and are certified by the same standards as hardware HSMs. So you can and certify at least as an issueing bank (and because of that, probably for a lot more).

This is an area of cryptography called whitebox cryptography. Tl;dr is that most solutions nowadays are proprietary, and all open sourced and documented whitebox cryptosystems are broken.