I found it easier to create a fine-grained github token with minimal scope/permissions for the tasks I intend to delegate.

For context, I'm running multiple agents and each of them is isolated with its own token.