What linker are you using? I've gotten really significant speedup over the default linker by using lld or mold.