A 1T model with quite few active parameters though.
A 1T model with more active parameters is obviously more expensive. If it had 1T active parameters it would probably be 20 times more expensive to train.
A 1T model with more active parameters is obviously more expensive. If it had 1T active parameters it would probably be 20 times more expensive to train.
This was pretrained for 2 months, so a model with 1T active parameters would take 3.33 years. Maybe a 4 months pretraining run can be acceptable in a future where Euclyd or VSORA machines have been rolled out in quantity. But we don't know how far off that is.
But that (i.e. models made with 4 month pretraining runs) would probably allow an EU-made genuinely China-beating open model, so if VSORA is rolled out in quantity, let's say, tomorrow, then this could be a thing that would happen this very spring.