Nothing stops a threat from just lying and waiting for you to expose a large number of passwords. Having one stolen doesn't raise red flags in itself.
I don't think signed addresses will be particularly effective. With the sort of key stores we have now, it seems pretty plausible that a bad actor to get a certificate that would pass on the Trezor device. It raises the barrier of entry a little though.