Hacker News Reader

I've embedded llama.cpp for a while now. There may be a point where it's not embeddable anymore, but for now it works great. My experience with the others is don't bother. The build systems and dependency management is truly awful.

Also, I'm obviously talking about porting here. You will need to tailor the code to your needs, thinking of it as a library. You cannot run an unmodified program.

View on HN

Can you elaborate a bit more why you chose llama.cpp? From their docs:

> "The main goal of llama.cpp is to run the LLaMA model using 4-bit integer quantization on a MacBook"

I'm not interested in running an LLM on embedded devices; the NN I want to deploy is really tiny and used for audio signal processing. It extracts information out of a small chunk of audio. The code needs to run on various embedded ARM chips.

What would be the advantage of using llama.cpp for this?

Reply on news.ycombinator.com