The way the language model is trained is by rewarding it for correctly predicting the next word in a sequence.

The output of the model is a predicted probability distribution of the next word and a “state” — the next iteration takes the state output of the previous interation and generates another word and state (and this process repeats many times).

Since there’s a probabilistic dimension, what may have happened in this case is that it happened to repeat once by chance and the model had learned that if something repeats 2x it’s likely that it will repeat a third, fourth, and fifth time.

Basically it’s just trying to game the loss function which rewarded it for predicting the next word in the sequence correctly.