> Maybe 10T? 100T?
I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.
I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.
If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.