How do you know? Android and iOS apps are much better sandboxed than desktop applications (which usually aren't really sandboxed at all). I would guess the chance of phones getting compromised is much lower than for the average desktop.
But that is besides the point, because a hardware token is almost impervious to attack. One would have to engineer a very specific bit of software or hardware and have physical access to this token. Requiring a phone app to confirm a payment made on your desktop just increases the attack vector.
The desktops I see on a daily basis don't have only a browser and office application installed.
I'm not sure which phone or OS you are using, but almost all smartphones have some kind of TPM or TEE nowadays. FWIW, the banking app on my Android phone makes use of that through the KeyChain API.