3-4 seconds is extremely fast if you're coming from the C/C++ world. I can't see how this could possibly be a roadblock.
That said, I expect incremental compilation built into rustc to greatly improve the situation - if done right, it should easily beat C++, although I don't know the details of the plan - so I'm happy someone is working on it. (In the past I wanted to work on it myself, but I was in a big generally unproductive slump..)
Although I agree with your point, I've been using Rust quite a lot lately and compilation time was never an issue so far.
But maybe my projects just haven't been big enough :)