Isn't this issue also present with other popular authentication systems like cookies and client side certificates? If you have a key, you can access the system. If you can copy the key, you can access the system from another location. It's simple as that.
People keep asking to have super smart locks that somehow recognize stolen keys, but how can they?