Hacker News Reader

Comment by mmoskal | Hacker News Reader

All the nvidia/torch stuff is "only" to make it run two orders of magnitude faster or so. For the actual principles behind LLMS, 500 lines seems quite accurate.

How long would it take to explain what an LLMs is doing to someone from 1800s? It is remarkably simple.

View on HN

The article talked about building a basic version of an LLM. Even basic versions like Karpathy's TinyStories ~100 million parameter models which manage to generate comprehensive English for a restricted vocabulary are trained for hours on an A100. If you want to skip the nvidia/torch stuff you'll be both training for days and having to write your own training code, which is not going to be just 500 lines. The principles are simple - but that is true for any important algorithm of the last 100 years. As opposed to LLMs, most of those can actually be done in a few hundreds of lines of code with no specialized hw and software stack required for it to be practical.

> Karpathy's TinyStories

Did you mean Karpathy's tinyllamas? [1][2]

Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4]

[1]: https://huggingface.co/karpathy/tinyllamas/tree/main

[2]: https://github.com/karpathy/llama2.c

[3]: https://www.microsoft.com/en-us/research/publication/tinysto...

[4]: https://arxiv.org/abs/2305.07759

Karpathy's tinyllamas are based on the tiny stories dataset. I could have phrased it better, sorry.

Reply on news.ycombinator.com