It obviously increases the time by 20 times given fixed hardware, it probably also makes the hyperparameter search more difficult and expensive, depending on how predictable the performance of your architecture is.

This was pretrained for 2 months, so a model with 1T active parameters would take 3.33 years. Maybe a 4 months pretraining run can be acceptable in a future where Euclyd or VSORA machines have been rolled out in quantity. But we don't know how far off that is.

But that (i.e. models made with 4 month pretraining runs) would probably allow an EU-made genuinely China-beating open model, so if VSORA is rolled out in quantity, let's say, tomorrow, then this could be a thing that would happen this very spring.