GitHub recently introduced fine-grained tokens that can be scoped to a single repository that might work for your use-case https://github.blog/2022-10-18-introducing-fine-grained-pers...
The new tokens are still in Beta, so there are some other limitations: for example, GitHub Packages do not support them yet, so you cannot use them in NPM/yarn to get your private packages hosted on GitHub.