You can outsource it to something like authy, which is still sms based, but gets disabled if you install their app. They also claim that they can detect number porting attacks, so that might be marginally better.
>Maybe some pseudonymous proof using cryptographic functions of modern passports could be used somehow without revealing real identity to the passport issuer too?
You can still lose your passport. It's less likely than losing your phone, but still. Also, to access the cryptographic functions of a passport, you probably need a NFC reader, which isn't exactly accessible.