Ollama serves models on an OpenAI compatible API directly on the inference machine, which most open source harness software will connect to. The inference machine has a name like inferencemachine.local on my home network so the Ollama URL looks like http://inference machine:11434/v1 to connect to any model I've downloaded.
For this part, are you using a tool-calling harness like Pi or Codex? Or did you sort of hack it together with prompts that tell it what json to emit?
I just got the rest of the “application layer” for my setup working yesterday, with a small set of evals. Auto-filing voice notes is what’s next for me.
https://github.com/chr15m/runprompt
That has a bunch of Python tools for things like adding to TODO list, notes, looking stuff up, etc.