For passwords you're right, but at least it prevents theft of every password all at once.
For Bitcoin hardware wallets like Trezor, IIRC they either do or will support BIP-70 "Payment Protocol" payment requests that are signed with an X.509 cert, allowing you to verify the request on the dongle's screen.