It's easiest to improve the codebase where and when you alter it for feature work. This is also a good control for yourself to stick to relevant refactorings. This should be your professional autonomy.
The remaining like, 5 to 10 percent of the time that you need 'dedicated' tasks, should be explained to and supported by the PM because he trusts you due to having a track record of delivering.
Of course, reality isn't always ideal