For bigger annex repos with lots of pointer files, I just disable the git-annex smudge filters. Consider whether smudge filters are requirement, or a convenience. The smudge filter interface does not scale that well at all.