Yeah, This can definatly be used for local models, but the problem is that most personal computers cannot host large LLMs and the cost is not cheaper than closed LLMs. But for organisations, local LLMs are a better choice.
Sliding window chunking, RAG, etc. seem more sophisticated than the other document LLM tools, so I would love to try this out if you ever add the ability to run LLMs locally!
Goes from 20 tokens per second to 15 tokens per second nearing the ~3k token context length, with similar quality in output to chatgpt 3.5.