Interesting, lots of models are already trained on MLM (masked language modelling), so you can iteratively add mask tokens in the middle but you need some rules around termination such as probability/length of mask tokens...

I'd be interested in how to convert models without retaining, or minimal retaining, I'd have thought it would just work with iterative mask tokens at least for models trained with MLM