The access is transient and you remain in ownership over the credentials and account, because the credentials were not compromised (just the programs/browsers with pre-existing auth). Though with physical access it's probably only a mater of time before local admin passwords are brute forced and access to keychain/browser saved logins is inevitable?
> If I steal your secure token
It's more like you just logged in using your secure token then someone stole the device and took advantage of that. I think we can all agree there's a difference between having access to the laptop and access to the account without the laptop.