This, however, doesn't apply to the code generation phase with LLVM which is already parallelizable at the codegen unit level (the number of codegen units is configurable), but that's the “back-end”, whereas this new parallelization applies to the “front-end” of the compiler.
Sort of, it's managed at the crate level and not file or module indeed, but the compiler then splits crates into smaller chunks called “codegen units”.
See: https://doc.rust-lang.org/rustc/codegen-options/index.html#c...
> This flag controls the maximum number of code generation units the crate is split into. […] When a crate is split into multiple codegen units, LLVM is able to process them in parallel. […] The default value is 16 for non-incremental builds. For incremental builds the default is 256 which allows caching to be more granular.
> This, however, doesn't apply to the code generation phase with LLVM which is already parallelizable at the codegen unit level (the number of codegen units is configurable), but that's the “back-end”, whereas this new parallelization applies to the “front-end” of the compiler.
> But won't this steal resources from the top-level file parallelism?
It won’t. rustc uses jobserver protocol to coordinate parallelism with cargo, so the total amount of threads doing compilation of the whole project doesn’t exceed CPU count.
> When you compile a Rust program, Cargo launches multiple rustc processes, compiling multiple crates in parallel.