The problem I deal with is that if I see something worth improving that doesn't get fixed now as I see it, unless it's a really significant issue, the code won't get improved down the line
At work, we often create tickets to deal with such things and have epics dedicated to "code gardening". These are often addressed on a Friday afternoon or at other times when one feels their productivity is especially low.