In theory the Linux kernel keyring would help here, even with a tsm or in conjunction with it.
Unfortunately as the industry abandoned the core Unix permission system (uid/gid) all of these methods just get a devfs[null] bind mount.
Only process that also support the traditional co-hosting model like nginx and Postgres do.
We would need nonce keys to gain no value from kernel memory or hardware storage.
You just need to tighten your sshd config, you can even add a "touch required" of the Yubikey to the sshd config. Has been in debian stable since like 11 I think?
So it's super friendly to integrate and very secure, as you need to physically be on your pc, have your yubikey and have your exact pc. So that's a lot of factors.