Everyone says this but I've yet for it to ever actually be a problem.
That's because it isn't unless you have a very exceptional case or maybe you make a mega-crate. I was rocking a severely outdated processor and it was slightly annoying, but not that bad at all. And once the dependencies are built it was a non-issue. With a more modern CPU the times are miniscule.
Right that's been my experience too. The initial build of dependencies takes a bit but that's never a problem for me personally.