6.5M! Impressive. Is it initialised or distilled from any base model or entirely from scratch? What kind of training data for the RAG and conversational chat specifically?
I've been surprised myself. For reference, it gets better ppl (and with faster training), than the nanogpt model from Karpathy's tinyshakespeare tutorial, at smaller size (nanogpt is 10M params)
Trained entirely from scratch. Which gets it pretty far in terms of basic conversational skills, especially when also trained on chat conversations
The training sources are:
1. Google Synthetic PersonaChat
2. DailyDialog
3. SODA conversational data
4. Hand-written model-personality conversations, repeated during training
5. Identity-corrected conversations—rewritten names, human-life claims, and hobby responses
6. 960 synthetic recovery conversations, teaching it to admit ignorance after user push-back
The final set contains 23,270 conversations
However, the chain reasoning and math problem solving skills, I was only able to add recently, by grafting certain parts from a 1B model, then doing space-weight programming, and finally finetuning/training to get the model to "get used to" reasoning. The "full model" is more of a harness + model:
* Frozen base model to interpret questions
* Additional trained heads to select question types, evidence, and follow-up behavior
* A library tool for retrieving facts
* Working memory and internal script language execution for arithmetic, conversions, and comparisons across answers. For example, it can retrieve a building’s height, remember that number, then execute “half of that”
* Identity extraction and physical simulation, so you can do thing like - User: "Ben has the ball", User: "Who has the ball", Model: "Ben", User: "Ben gave the ball to Sally", User: "Who has the ball now?", Model: "Sally"
It's relatively easy adding skills either via its internal scripting language, grafting+weight programming, fine-tuning and/or training it to use a new skill. The main limitation right now seems to be just quality data and proper supervision, ie. someone to talk to the model, flag issues, fix them, then re-test. This is kinda tedious, but I've found that even the best frontier model is kinda lacking in quality when producing synthetic conversational data, and most datasets out there have a lot of cruft. So I get better results just manually producing the data and iterating, than feeding it GBs of public datasets (even famous datasets have a quite a bit of un-reviewed data, as well as errors, which aren't even published)
Creating and training this model has been a lot of fun. Since it's so tiny, it's easy to experiment with and try a lot of different things. Most training I can just do in a few minutes locally