Great work! I've been playing a little with an older version of this (from ~ March 29th, when it was called `llama-rs`) just to get a taste of the LLM world, and it's super convenient. (I haven't tried llama.cpp to compare.)

Since it's a bit slow on my hardware (being CPU-only at the moment + me not having any fancy graphics cards, anyway), when toying around I sometimes get impatient and fire up jobs on multiple servers via the CLI. I keep toying with the idea of whipping up a webapp that would provide a ChatGPT-like interface and dispatch to any number of connected backend servers.