No llama.cpp nor any compilation complexity. Run with two Python commands!
llama.cpp will remain a single two line project (git clone https://github.com/ggerganov/llama.cpp, make -j) that will compile easily and run on anything. No external deps to pin to a particular commit (that will only have a lifetime of some months) as things change rapidly.
That said, the changes in the ggml weights format the last 2 weeks were annoying, but now that the mmap-style weights are settled on it should be less converting. In that sense huggingface wins, it only has two incompatible weights formats. llama.cpp's ggml has had 3.
also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.
Also, hi? I don't recall you by that nick but the internet is a small place sometimes.
specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the learning curve.
did our channel survive the demise of freenode? I was andares, I think I used to be annoying but I've gotten better.
(The irony of holding this opinion while dealing with pre-trained AI models is not lost)
And, PyTorch does so well on GPUs!
Afaict from the docs, you still need to request the original Llama weights from Meta (or get ahold of them another way), then apply the diff-weights requiring 60GB RAM?