https://arxiv.org/abs/2305.07759
> In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.
Because they don't work very well in real life. Article doesn't have any experimental proof it works, saves training time, or size, or anything..
And it doesn't mention another promising approach: mix of experts. There are many ways of implementing it. I mean the general idea of specialized fragments of the NN which are selectively called. They don't have to be trained all at once.
Even the layer duplication trick they were using doesn’t work. I tried a refinement of that where you fine tuned the activations of the added layers to match the original activations which helped a bit relative to what DeepMind found but it still wasn’t material enough to be worth bothering with
One thing where additive training works is fine tuning, like LoRA. But that's different. It's like poor man's way. For those who cannot train the whole model. Most likely uptrainig it whole is more efficient, just my guess.