In the Sublime version, one of the things I did to avoid that is it doesn't call `git diff` if it has been less then 5 seconds since the last `git diff`.

Or a reimplementation of the algorithm in VimL ;)