I have been doing this for a little over a year now. The most difficult/frustrating thing was using it on my phone. Luckily, TermBot [0] solved that, allowing me to choose an external agent for my key with OpenKeyChain [1]. A Yubikey 5 with NFC and I feel real futuristic tapping my phone to my pocket to gain access to remote computers. I get to do this with my password manager too [2].
[0] https://f-droid.org/en/packages/org.sufficientlysecure.termb...
[1] https://f-droid.org/en/packages/org.sufficientlysecure.keych...