Except then there's no parallelism since you're only building one file. Ideally you'd split it into N files to take advantage of multiple cores, but then you have to decide how to split it...
And each system should only have a single 'public interface header' to keep the number of cross-system include dependencies low.