I've been running Ollama in my home lab for awhile now but I wanted to experiment with running something a little more "low level". I saw a ton of posts about Llama.cpp, which looked interesting, but there wasn't a lot specifically about running vLLM on Proxmox, so I thought I'd give it a try. Setting up vLLM in an LXC wasn’t necessarily difficult but, even after doing it several times, it was still tedious. These are the notes I’ve taken along the way, if only for my own reference next weekend.
Feel free to tell me what I got wrong :)