So people invented cold/hot wallets, like, 10 years ago.
Store most of the fortune encrypted and out of internet connection or on hardwallet, if you have large sums, transfer small sums (like, monthly usage) once in a while to your phone wallet and use freely.
It's not that easy to loose access, by the way, if are not dumb and don't open all shady executables you could find in the internet.