Are you using stock magit? I think a few of the GitHub integration plugins for example have performance issues, so it may not be Magit's fault but some of the other extension you're using.