If you’re looking at The Pile, you also might consider the Red Pajama dataset. A new cleaned version was released recently https://www.cerebras.net/blog/slimpajama-a-627b-token-cleane...