Hacker News Reader

Comment by echelon | Hacker News Reader

Looks like they raised funding though. They're hiring.

Side-note: I really don't get the C++ in AI thing at all. It's becoming a meme. I wonder why they didn't go with Rust instead. It'll be easier to deploy to the edge as WASM, too.

View on HN

Although Rust is an amazing language with rapid adoption, it still has a relatively smaller user base in low-level programming, where llama.cpp operates. As a result, the pool of talent that can contribute to such projects would be more limited. Almost all low-level programmers can write in C++ if needed, as for C programmers, it's essentially like writing C with classes. Rust programmers, especially those who care about low-level details including throughput and latency, almost always have a background in C or C++. If llama.cpp were written in Rust, there would likely be far fewer contributors. Considering that one needs to be at least interested in deep learning to contribute, the fact that it currently has 476 contributors is impressive. [1] I think this is one the most important reasons the project can move so fast and be such an essential project in the LLM scene.

[1]: https://github.com/ggerganov/llama.cpp/graphs/contributors

Llama.cpp is a lot of C SIMD. I think using C++ elsewhere in the project just made sense in its infancy, when it was CPU only.

Rust is really interesting for stuff besides running the actual llm though. Python can be a huge performance/debugging pain when your frontend and such get huge.

> Rust is really interesting for stuff besides running the actual llm though.

That's the reason why llama.cpp is so attractive though. It just runs the LLM.

Thats not giving the project enough credit though, it implements a ton of very useful features outside the llm itself.

My understanding is the proliferation of “XYZ-cpp” AI frameworks is due to the c++ support in Apple’s gpu library ‘Metal’, and the popularity of apple silicon for inference (and there are a few technical reasons for this): https://developer.apple.com/metal/cpp/

Because they didn't write llama.cpp.

What does Rust offer here? More people know C++, security stuff from rust comes in the way of rapid exploratory development and arguably, C++ templates (don't think they are used here, moot point I know) are much more superior to rust.

Seriously? 'But rust'.

Reply on news.ycombinator.com