ag has issues with .gitignore, as patterns are not recognized properly. The matcher that git uses cannot be included due to licensing (see https://github.com/ggreer/the_silver_searcher/pull/614).
Does seem like there is hope, as a BSD licensed version exists. My fingers are crossed that this is solved soon :)