I encourage you to fork this repo and build it.

Might be worth checking out Ollama and bakllava. https://ollama.ai/library/bakllava

Maybe the model is a bit too slow, but I'm sure smaller ones will come out soon. You can likely fine tune to do exactly what you need.