Yes. Continuously.

There's 3 times to refactor:

1. When you're trying to add a new feature ("prepare").

2. When the new functionality has been added ("tidy").

3. When you're trying to understand the code ("clarify").

The problem is when refactoring is seen as a separate activity from coding. It shouldn't be: it is a _part_ of the coding activity.