I think that Git submodules are to blame. I think that the idea is great, but the implementation is cumbersome. If they were better to use it would solve a lot of these problems.
Ultimately the problem is in scaling the number of build inputs, not the number of .git directories.