He had a singular way of revising, almost pathologically, holding some key points utterly fixed and revamping everything else to make them fit. It's why he never finalized The Silmarillion, and why when his son finally did publish it, he discarded a lot of the work done post-Lord of the Rings.

I believe you are correct: Tom was one of those fixed points, for no reason he could explain. He later said that Tom was left as a deliberate mystery. Which is a bit disingenuous, but that's fine. Tolkien worked by instinct, and his works are extraordinary. If his instinct tells him that Tom needed to be in the narrative, that's fine by me.

If other readers are perplexed by it, they will hopefully forgive it long enough to get through the book and read it again. It's very much the kind of book that you won't fully understand in a single reading, if ever.