Highly recommend lemonade server if you have a strix halo desktop. Been using Qwen3.6-35B @ Q_8 as my main driver and it’s been great with 60 TPS for generation. I occasionally use the 27B @ q6 but only get 20-25 TPS for generation with MTP.
I used llama swap for a while before leaning into lemonade. The UI has improved a lot, but be careful as the most of the models default to very small 4K context windows by default.
They’re doing some nice things with their Halo models, which load an ensemble of different types of model at the same time. With high vram it’s easy to keep them all in memory, so even though the compute is limited the context switching is fast.
You do lag a bit on the upstream engine releases, the llama.cpp/sd.cpp/whisper libraries are downloaded from inside the app.
vLLM is in experimental mode, I haven’t tested it. It’s limited in the models they suggest, but you can download anything from huggingface with a two click install.
Is there some means to monitor the queries it's sending (or hold and review before transmission) or throttle to avoid triggering abuse thresholds on any single domain?
I think it stores basic logs but if you wanted to monitor you’re probably going to want to proxy it with an LLM gateway.