Harbor is a toolkit for working with LLMs locally, it got an upgrade recently - an LLM proxy (called boost).

It comes with first-class support for streaming, multiple downstream APIs and custom modules, aka scripts you can write to bash some workflows together.

One of the most interesting built-in modules is klmbr, which can inject entropy into your prompts, making LLM outputs much more diverse as a result.