It is self-serving and dangerous to ascribe irrationality to behavior that is out of the norm from our perspective. The problem is that in many cases, what we deem as irrational behavior is simply behavior predicated from different beliefs or different values. How Putin evaluates different outcomes for Russia is likely to be very different than that from a western context. Our strategy to avoid a nuclear war must be in light of our assessment of his beliefs and values, and an analysis of how he will maximize Russia's outcome from that perspective.
If our policy is one of complete Russian retreat, and we reason from our context that escalating to nuclear weapons to secure victory is irrational, we're all but assuring an eventual nuclear confrontation with Russia. The only way to surely avoid nuclear war is to allow Russia an avenue to some kind of victory in Ukraine using only conventional weapons. We want to severely raise the cost of this victory, but at the same time we should accept the outcome if Russia is willing to pay it. If our policy is such that the choice is left to Russia to either go nuclear or completely withdraw, it will just represent the complete recklessness that our policy towards Russia continues to be.