Your security recommendation is bad.
> password manager on an encrypted thumbdrive, along with your gpg key and private ssh key with the decryption password + password manager password being the only ones you remember
You have to give the system both passwords to use the thumb drive, except that these passwords, rather than being compartmentalized as currently is the practice, can steal literally every key I need to have portable with me.
By contrast, if I just type in a single password to the system, it can only even potentially the system to which that password corresponds.
Your system drastically increases the failure modes of the whole system, from a single password to a known edge server, which can be locked down on the server side, to a potential compromise of every password.
> The best mitigation is to only remember passwords that are only useful in conjunction with physical access (the password manager / decryption ones).
This is extremely bad advice: the best mitigation is to require two factors, not a single factor. Your strategy is inherently two single factor systems, not a two factor system. This is what gives its ability for cascades of compromise.
The correct way to do this would be to have the keys corresponding to each server represent an XOR of your password for that server with a base key stored on a physical device, combining the two pieces of information to generate the final key. This reduces the effects from the compromise of your thumb drive from "full compromise of all systems" to "reduced to password based authentication".
Please be more careful in your recommendations in the future.