This huge language model was trained 'from scratch' - ie. before the first batch of data went into the training process, the model state was simply initialized using random noise.

I believe we are near the end of that. As models get more and more expensive to train, we'll see future huge models being 'seeded' with weights from previous models. Eventually nation-state levels of effort will be used to further train such networks to then distribute results to industry to use.

A whole industry will be built around licensing 'seeds' to build ML models on - you'll have to pay fees to all the 'ancestors' of models you use.