Llama.http: A Dockerized, simple HTTP API for llama.cpp

Simple & straightforward, it's a quick spare-time project designed to enable automation and not as a standalone way to publicly host the models (no rate limiting, persistence or auth). The project README is here: https://github.com/edfletcher/llama.http/tree/master/example...

With the Docker image, it's just a matter of grabbing the model binaries you want, adding a small sidecar JSON [1] describing them and you're up and running.

Really just a porting/re-targeting onto the parent llama.cpp project of the HTTP I'd built on the alpaca.cpp fork awhile back [2] in order to take advantage of all the great new GGMLv3-encoded models, mmap'ed loading, etc.

All I'd ask if you build anything neat on top of it, please share it so I can use it too! Thanks & enjoy

[1] https://github.com/edfletcher/llama.http/tree/master/example...

[2] https://news.ycombinator.com/item?id=35417730