We need to recognise that is impossible. Making getting into a bank account easier for the owner does so for everyone else too. And also makes it easier for the bank to ditch the secure fallback - a building.
And ask why vendor-indepedent trust shops aren't sprouting in every locale. Answer: same security problems.
We are well and truly doomed to living in a scammers' paradise.