How does this differ from SuperBPE, which seems to pursue a similar goal? https://arxiv.org/abs/2503.13423
Looks like parallel invention. (I’m not associated with the paper or its authors.)
Looks like parallel invention. (I’m not associated with the paper or its authors.)
In Boundless BPE, no schedule must be chosen, because there is not any point at which the constraints of pretokenization are removed entirely. Instead, at any point in the learning process, merges between adjacent pretokens are permitted if the pretokens are each represented by a single token. There are some additional details about how the authors incorporate Picky BPE, which I will not try to repeat because I would probably get them wrong.