They trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens while staying under a given compute budget. The results are modelled, and they pick the best one. Turns out the having a bit fewer tokens improves performance.