I'm a bit curious what you're working on, and if there might be some interesting connections there. Would you like to speak? You can just book in my calendar through the site.
Anyway:
It's an open-source licensed, distributed data orchestration framework designed from the ground up for HITL workflows where correctness matters more than speed (primary field is medicine, but law, etc. could also benefit). It sounds like we're attacking the same problem from complementary angles. You're building the human routing API, I'm building the pipeline infrastructure that defines when and how humans get routed into the loop.
The core idea: pipelines are YAML-defined state machines with explicit correction steps. When a worker (e.g. your LLM endpoint) produces output, the pipeline can pause, send results to a human reviewer, and wait for either approval or corrected data; all as first-class custom protocol messages (based on Majordomo protocol). The correction protocol has timeout handling, strike counting for repeated failures, and an audit trail that captures every decision point. Also, the YAML can define how to "steer" the pipeline in case of a correction, it can continue, store the correction, route to a specific step, fail, etc. (combinations also possible, e.g. store the correction, continue or jump to another step). A feature creep that's currently itching is implementing a largely reduced Lucid (the dataflow language) syntax set parser and transpiler into YAML pipeline definitions.
What might interest you: every message in a pipeline shares a UUID, and each correction creates an immutable record of what was changed and why. This is essentially your "structured training data" as a sort of (useful) byproduct of the architecture: you don't extract it after the fact, it's the communication protocol itself. Its intended workflow philosophy is an iterative fine tuning, I guess, with training data for
The framework uses ZeroMQ for binary messaging (sub-millisecond routing overhead) and can run from edge devices to datacenters. If it speaks TCP/IP and can run Python 3.11+, you can plug it in. Workers are pluggable, your existing model endpoints could be wrapped as the framework's workers with about 20 lines of Python, receiving tasks and returning results through the same correction-aware pipeline. All the components of the framework have lifecycle aware "hooks" so when you design your workers for example, in Python, you define them as a class and decorate their methods with @hook("async_init") or @hook("process_message") and those hooks get executed at each lifecycle event.
So in your project, instead of clients defining pipelines on their end and querying your API, the framework could provide the orchestration layer that routes between your clients' models, your human review queue, and back—with the pipeline definition living in a YAML file rather than scattered across client code. Your humans would interact with a well-defined correction protocol rather than ad-hoc intervention.
No HTTP endpoint (yet), you'd need to implement a worker that relays e.g. REST API calls and translates them into the framework's messages.
It's LGPL-licensed, intended for federated machine learning and self-hosted scenarios, and the (initial, now fairly more complex) "spartanic philosophy" means the core stays minimal while complexity lives in pluggable workers.
But it's not MVP ready, some things are still broken and I'm trying to hit the 0.1.0 version with a simple demo that takes a WAV file, transcribes it into text, then another model extracts keywords from the text, including intent and the basic context, then it all goes to another model that generates a TinkerPop/Gremlin query based on it, then the client executes the query and the results get sent along to the final worker that summarizes the (reduced) knowledge graph. That'd show a multi modal pipeline in action.
If you're interested, find me on github, the username is the same.