Where do you store the passphrase of your passphrase database ?
Of course, that assumes the master password is chosen such that it isn't susceptible to your typical dictionary/hybrid-dictionary attacks.
Your memory is a SPOF, and while generally robust for this sort of thing, it's not bulletproof.
neos can store the rsa key and use it to authenticate via ssh, as it has opengpg smartcard emulation support