Refactoring as a separate task is a bad idea from good intentions. If you blow your schedule on some other higher-priority tasks (i.e., directly business-relevant), guess which task is the top candidate for deferral? Speaking from experience.
Paying down technical debt is business-relevant, but not directly relevant. That means things on fire or new contracts or whatever will always supersede things of indirect benefit. Under sufficient load, this effectively means no refactoring, because people are then trained to wait until they are tasked with "the refactoring task".
What has served me well is following Fowler's advice, which is repeated in this article. Refactor when you want to do something else. I would add: Prefer modifying only code that will be QA tested anyway, so you don't explode your testing surface area. Do it as part of a directly business-relevant task. Build it into your estimates.