Multi-Token Prediction Done Right

arxiv.org

13 points·matt_d··4 comments

This is such an elegant paper. At the risk of minimizing the thought that went into it, I love how it boils down to cleverly introducing zero, twice (to decompose the hard MTP gradient and to show the effect of soft MTP on the gradient's bias and variance).

I wish I could come up with this kind of stuff, haha. Clearly came from spending some time looking carefully at and thinking carefully about MTP heads. Kudos and thanks for an interesting paper!

thank you for this comment! it was really a serendipitous idea—i was playing around with amortizing the KL computation in our previous Next-Latent Prediction work (https://arxiv.org/abs/2511.05963) and this Soft-MTP idea trickled out from it. i sincerely appreciate your kind words, it means a lot to me!

The results look like a statistical fluke to me. Tiny effect, sometimes negative.

hey thanks for the comment, author here. That's a fair read of the loss numbers, I could only fund up to 1.2B scale myself and I don't want to oversell the results. But note that the benefits that Soft-MTP delivers in the paper is *consistently growing across scale*. The residual is irreducible with scale and will only become noisier as model grows bigger, so removing it via Soft-MTP is intuitive. It is also easily verifiable cos it only needs minor code change, codebase: https://github.com/JaydenTeoh/Soft-MTP

there is reproduction of Soft-MTP's benefit over standard MTP at 35B scale but it is only one data point for now: https://x.com/HivazeMe/status/2118432486724391217

Reply on news.ycombinator.com