Given the author hand-rolled his own FFT, I'm also guessing it's not as performant?
In comparison I could just type git clone https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU acceleration working with my AMD RX 580 8GB. Plus with the llama.cpp CPU mmap stuff I can run multiple LLM IRC bot processes using the same model all sharing the RAM representation for free. Are there even ways to run 2 or 3 bit models in pytorch implementations like llama.cpp can do? It's pretty rad I could run a 65B llama in 27 GB of RAM on my 32GB RAM system (and still get better perplexity than 30B 8 bit).
You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different language and proclaiming that an entire language/ecosystem is crap.
The only real take away is that many projects, independent of language, put way too little effort towards onboarding users.
Python is a mess. llama.cpp was literally a git clone followed by "cd llama.cpp && make && ./main" - I can recite the commands from memory and I haven't done any C/C++ development in a long time.
I know exactly what you mean, but I'm probably so inured to it by now that I've just come to accept it. Obviously not everyone feels this way!
The example you gave works because llama.cpp specifically strives to have no dependencies. But this is not an intrinsically useful goal; there's a reason software libraries were invented. I always have fun when I find out that the thing I'm trying to compile needs -std=C++26 and glibc 3.0, and I'm running on a HPC cluster where I can't use a system level package manager, and I don't want to be arsed to dockerize every small thing I want to run.
For scientific and ML uses, conda has basically solved the whole "python packaging is a mess" that people seem to still complain about, at least on the end-user side. Sure, conda is slow as hell but there's a drop in replacement (mamba) that solves that issue.
Coming from rust (and nodejs before that), the package management situation in python feels like a mess. It’s barely better than C and C++ - both of which are also a disaster. (Make? Autotools? CMake? Use vendored dependencies? System dependencies? Which openblas package should I install from apt? Are any of them recent enough? Kill me.)
Node: npm install. npm start.
Rust: cargo run. Cargo run —-release.
I don’t want to pick from 18 flavours of “virtual environments” that I have to remember how to to “activate”. And I don’t want to deal with transitive dependency conflicts, and I don’t want to be wading through my distro’s packages to figure out how to manually install dependencies.
I just want to run the program. Python and C both make that much more difficult than it needs to be.
Yeah, no… my current $DAYJOB has a confusing mix of nx, pnpm, npm commands in multiple projects. Python is bad but node is absolutely not a good example.
It’s just that almost nobody does that.
What we want instead is to combine typescript, js bundlers, wasm, es modules and node packages all together to make web apps. And that’s more than enough to bring seasoned engineers to tears. Let alone adding in svelte / solidjs compilation on top of all that. I have sweats just thinking about it.
Rose colored glasses if you ask me. The difference is it seems you use Node often (daily?) and have rationalized the pain. Same goes for everyone defending Python (which I’m sorta in that camp, full disclosure), they are just used to the worts and have muscle memory for how to work around them just like you seem to be able to do with Node.
It’s not just that. Node can also handle arbitrary dependency graphs, including transitive dependency conflicts, sibling dependencies and dev dependencies. And node doesn’t need symlink hacks to find locally installed dependencies. Node packages have also been expected to maintain semver compatibility from inception. They’re usually pretty good about it.
It’s not perfect, but it’s pretty nice.
In general solving such environments (where some versions are left floating or only have like >= requirements) is an NP-hard problem. And it requires care and has to draw source code and/or binaries from various sources.
This is the problem that conda/mamba solves.
If you just want to install python packages, use pip+virtualenv. It's officially supported by python. And while pip has traditionally been a mess, there's been a bunch of active development lately, the major version number has gone from like 9.xx to 23.xx in like the past two or three years. They are trying, better late than never, especially for an ecosystem as developed as python's.
So, if you want to compare rust/cargo, and it handles non-rust deps, then the equivalent is conda. Otherwise, it's virtualenv+pip. I don't think there are any other serious options, but I agree that two is not necessarily better than one in this case. Not defending this state of affairs, just pointing out which would be the relevant comparison to rust.
You can run arbitrary code at build time, and you can link to external libraries. For the likes of C libraries, you’ll typically bind to the library and either require that it be installed locally already, or bundle its source and compile it yourself (most commonly with the `cc` crate), or support both techniques (probably via a feature flag). The libsqlite3-sys crate is a fairly complex example of offering both, if you happen to want to look at an actual build.rs and Cargo.toml and the associated files.
> [pip’s] major version number has gone from like 9.xx to 23.xx in like the past two or three years.
It skipped from 10.0.1 to 18.0 in mid-2018, and the “major” part has since corresponded to the year it was released in, minus 2000. (Look through the table of contents in https://pip.pypa.io/en/stable/news/ to see all this easily.)
I understand why people find Python's packaging story painful, I guess is what I'm saying.
mamba can do this much quicker.
No need to denigrate those who don't need to import all their code; it's perfectly fine to not rely on third parties when developing software - some of our best works result from this.
A self contained zero dependency Python project is also literally “python app.py”
The barriers to entry are lower with python, and so I think there tends to be a lower standard for deployment.
It's annoying to see them "glueys" compared to the binary compiled segment where the heavy lifting is done. Python and others exist to latch on and assimilate. Resistance is futile:
https://pypi.org/project/pyllamacpp/
https://www.npmjs.com/package/llama-node
Deploying a PyTorch running Python app to anything in a way a user can just run it is a struggle. Even iOS aside that’s not a small use case that’s all local and offline ML potential.
Really makes me wonder about the whole language, did they ever expect the code to have to run elsewhere than the machines the writer controls.