I agree that benefits come from fine-grained permissions; however, for the fine permissions of user authentication (and API authentication), I think it would help to use X.509 certificate chains. The certificate can include a extension for specifying the permissions that it grants (a permission will only be granted if all certificates in the chain grant that permission). This avoids needing to login or use 2FA in order to set up new API keys or access tokens, and prevents problems with misauthentication with the wrong server (personal access tokens don't help much compared with passwords, but X.509 does help). Merely using X.509 won't cause it to have fine-grained permissions, but they can be combined in the way that I described. (This use of X.509 can also be used to allow someone else to operate on your behalf with limited permissions; in this case, a client would issue a certificate to a server, and that server would then also be a client to GitHub.)

If there is a broken design of GitHub Actions, that is another issue, and neither TLS nor X.509 is the issue with that. (Also, GitHub Actions could hopefully be made to allow finer permissions if the existing permissions are not fine enough; that is not related to X.509 at all.)

Avoiding too many dependencies, is another thing to do.

And, signed releases is another thing to do, and is helpful for additional reasons (unrelated to authentication with GitHub, but useful for knowing that you published it rather than someone else (without having to trust GitHub with it)). The key used to sign the release might or might not match that in the X.509 certificate used to authenticate with TLS; there are some benefits if it does (such as if you have published your certificates that someone else can check, or your mention of Yubikey), but it doesn't have to (e.g. because you have multiple certificates, because multiple people are publishing releases, because you have some other reason to not use the same certificate for the other purpose, etc).