I think it's less a LoC thing than it is a number of symbols/files thing. We have some 900k+ tests with an accompanying large number of test asset files. That means 900k public methods that test discovery has to process in realtime.
It's partially on us to move away from the antipattern of solution monoliths, and to nugetize crap that doesn't need to actually be in the solution. But it's also on Microsoft to lazily load projects that aren't being used.