I have had this conversation yesterday, and the refactorer asked this very question. Here's my answer:
The question is improperly formulated. With your behavioral patterns of immediate refactoring, the question seems to be: "if I don't refactor this code NOW, who will?".
I'd say that if you don't refactor the code now, someone else, maybe future you, may refactor it later.
Most sane companies will protect engineering from impulsive feature requests and will add them to the backlog for future consideration.
I'm trying to instill the same protective mindset in my teams, and help them not do refactorings immediately but rather consider them as potential work as dedicated items in the sprints or so.
Doing so, prioritizing becomes a natural part of the refactoring/tech debt/tech risks agenda, because we have an overview of the work to do globally.