I don't see any access granted to Pokemon Go (it's not even listed) in the "Apps Connected to your Account" page: https://security.google.com/settings/security/permissions
I am running on a Nexus 6 and signed in with my Google Account when I first launched the app.
Try revoking access and see what happens. Worst case, it might ask you to sign in again.