The training data back then was sooooo bad. You'd open something at random in Common Crawl and it was truly unbelievable that models (like llama) were able to output coherent stuff. The architecture was truly miraculous if it could train well with terabytes of garbage.

There was so much low hanging fruit back then in terms of training.