Here is the commit that added ROCm support to llama.cpp back in August:
https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
I had wrapped this and had it working in Ollama months ago as well: https://github.com/ollama/ollama/pull/814. I don't use Ollama anymore, but I really like the way they handle device memory allocation dynamically, I think they were the first to do this well.
- what's special about the memory allocation, and how might it help me?
- what are you now using instead of ollama?
I'm mainly just using llama.cpp as a native library now, mainly for the direct access to more of llama's data structures, and because I have a sort of unique sampler setup.