Given some of your other comments here, you got me curious. So you don't want it to access you email, then what email are you hoping for it to access? If it's using its own siloed calender, how would you ask it "am I free tomorrow?"
So your needs would be roughly met by creating a new email under your business domain (like you would for a new employee: agent@example.com, let's assume you're using Google gsuite), and then using chatgpt connectors with that inbox and gdrive. I understand you don't want to set all of that manually, but is that final state close to what you're envisioning?