This is a common misapprehension. Actually, even if you fully mock out time, you can still get race conditions, because goroutines can remain active regardless of the state of the clock, and there's no general way to wait until all goroutines are quiescent waiting on the clock. This is not just a theoretical concern - this kind of problem is not uncommon in practice.
I think clock-mocking can be very useful for testing hard-to-reach places in leaf packages. But at a higher level, I think it can end up producing extremely fragile tests that depend intimately on implementation details of packages that the tests should not be concerned about at all. In these cases, I've come to prefer configuring short time intervals and polling for desired state as being the lesser of two evils.