If you got drugged and someone took your phone to do a bank transfer, I would imagine there would be some hope of reversing the transfer, with a whole lot of painful steps. With crypto it's pretty futile if they manage to move it.
Also the $5 wrench attack can evolve, right? Just because you have your keys in different places doesn't mean you can be coerced into getting them together.