Hacker News Reader

This has been the dynamics with LLMs for awhile. The majority of LLMs are massively undertrained. 7b models are the least "undertrained" mainstream models we have, hence why they have proliferated so much among the LLM fine-tuning community.

Yes, it's not easy to train a 33B model. An interesting point is, naive fine-tuning, which means if one followed the standard way to fine-tune the model. Training a larger model is tricky, not only the data amount matters, everything like data cleaning, learning rate, and decays will affect the final performance.

> on ~1 million C programs, compared to [...] 2 trillion tokens, which is a few orders of magnitude more data.

Is that comparable like that? This would assume that the average C program of the set is orders (plural) of magnitude less than 2m tokens in size, which could indeed be true but sounds like an optimistic assumption.