Was thinking the same, so 1T might bring you to the league of GPT-4.
Actually in the best case, since it seems that meta, google, openai and so on have the most talent.
Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?
Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?
I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.
If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.