You don't have to, there is a world of things you can do with them without giving them access to your email or bank account.

"Hermes, clone this android app to my computer, add this and this feature and fix this annoyance and then rebuild it and push it to my phone"

"Hermes summarize this YouTube podcast and push the summary and transcript to my Obsidian vault"

"Hermes check mg obsidian notes for when I last wrote a blog article about Neovim's integration with AI agents"

*Hermes, you have an API token that allows read only access to the Zabbix monitoring system, check that and then use the Proxmox API token that only allows you to do limited actions to reboot the VM that is having issues"

Etc, etc,