Musk’s recent stock purchase and discussion about joining the board made the price jump significantly. You’re right that there’s no obvious reason that his involvement should change Twitter’s value, but the market appears to think otherwise. If he walks away, at a minimum, I would expect that the stock price will drop to what it was before he started playing around.
But if he walks away, I believe he’ll also do something to make the stock drop more. For instance, he’ll stop Tweeting, which should have a measurable impact on the number of people actually using Twitter regularly.