* https://github.com/jmorganca/ollama/blob/main/docs/api.md
* https://github.com/ggerganov/llama.cpp/blob/master/examples/...
Some like c0sogi/llama-api are pretty neat because they support concurrency, and supports multiple backends (llama.cpp and Exllama, although it could be expanded).
While you might lose out on some low-level configurability, being able to easily swap between OpenAI and local models is a big win in my book.
To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g. Chat). This likely can be done already by specifying the model parameter in Ollama (and others) but I've not explored it much yet.
https://langroid.github.io/langroid/blog/2023/09/19/language...