The game theory works roughly like this: Putin wins the most if his threats work. If they don't work, he has the choice between drawing a new red line, launching a limited nuclear attack on Ukraine, or full-on nuclear war. The last option is obviously out (guaranteed destruction is in no proportion to the reward). A limited nuclear attack would likely make him a pariah on the world stage and make it even more difficult to find trading partners. So instead he redraws the line.
If he ever went for the limited nuclear attack option Ukraine might give up, and the west would have to choose between war against Russia (high risk, low reward) or just tightening economic sanctions to the max and punishing anyone who dares to trade with him.
If we assume rational actors I don't see how this would ever escalate to nuclear war. And as long as Ukraine doesn't find a way to decisively push Russian troops out Russia has no incentive to climb the escalation ladder. Even as prolonged war leads to internal instability