difference is that if they keep something from my suitcase, I will know. I have no such guarantees when they access my phone unless I see what they are doing.

That means, that you should expect your device and all access tokens on that device compromised.