Purely for the SSH part if you’re uncomfortable using anything outside of OpenSSH itself for authentication you could consider certificates[^1]
You’d still need something to sign the certificates based on some other identity of course (it can be done manually but kind of defeats the purpose) be it smallstep or something else