I think that doesn't actually remove you from the GitHub contributors list. I had used the wrong identity for a commit and tried all manners of git manipulation and force push, but GitHub kept me on the list. In the end I just nuked that account.
Edit: Apparently renaming branches and reverting that works: https://stackoverflow.com/a/69294044