You can configure the agent to confirm each key usage to have your cake and eat it too. :)

It's also good to see if any malicious process tries to make use of the agent locally!