Definitely sounds like a problem. Do you think an alternative source control solution would handle that situation better? At some point huge with a large number of small files takes its toll.
It would be interesting for someone inside Microsoft to write a blog describing why Git works well for the Linux Kernel but not for the Windows Kernel. They cannot be that different from a layout perspective, can they?