Ask HN: Why the LLaMA code base is so short

3 pointskureikain2 comments

I was getting into LLM and I pick up some projects. I tried to dive into the code to see what is secret sauce.

But the code is so short to the point there is nothing to really read.

https://github.com/facebookresearch/llama

I then proceed to check https://github.com/mistralai/mistral-src and suprsingly it's same.

What is exactly those codebases? It feels like just download the models.

Most neural networks are just directed graphs, with a ton of matrix multiplies and nonlinear functions at the end of each layer. The libraries to do gradient descent, training, etc.. are all there to use. It is amazing how small the actual code is, compared to the amount of compute to do the training.

These are just the repos that provide the inference code to run the model, it requires the weights which are available via HuggingFace or in Llama 2's case, from here: https://ai.meta.com/resources/models-and-libraries/llama-dow...

Reply on news.ycombinator.com