Hacker News Reader

What is the currently favored alternative for simply running 1-2 models locally, exposed via an API? One big advantage of Ollama seems to be that they provide fully configured models, so I don't have to fiddle with stop words, etc.

View on HN

I just use llama.cpp server. It works really well. Some people recommend llama-swap or kobold but I never tried them.

llama-swap if you need more than 1 model. It wraps llama.cpp, and has a Docker container version that is pretty easy to work with.

Reply on news.ycombinator.com