Hacker News Reader

Comment by justusthane | Hacker News Reader

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-trained model locally? Surely the specs required to run a model are much, much lower than those required to train the model?

View on HN

I believe the training is where the architecture of the model is most apparent. You can absolutely download plenty of pre-trained models.

You will also probably need to fine tune for a specific use case, so a common approach is downloading a pre-trained model and fine tuning.

I think including the “from scratch” tuning script is educational more than anything else.

It's the equivalent of building from source versus downloading a compiled binary.

Also you can perform "fine tuning" which means you start with a trained model and train it further on your own data, allowing you to customize the model for specific tasks.

If you're only using pre-trained models, it's going to be harder to differentiate yourself. Training / specialization of models is where the moat-building is (due to access to different data sets / better ideas). By specializing / training, more of the token limit can be used for generation rather than prompting / better prompts can be made.

The lower the cost of training, the more profitable any resultant business. You can even envision businesses that train the model regularly to bring in new knowledge. The cheaper this is, the more opportunities open up.

inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?

Language models range from 1 to 300+ GB when loaded. It depends on how you load them, if you load in int8 you get 4x reduction.

Reply on news.ycombinator.com