I had a similar problem until today when I saw in the release notes you can turn off the gravatar icons that appear in each of the commits.

Doing this I found that the entire interface was much more responsive.