local running would be impossible as you would need your computer running 24/7 (since the widget is available on your website which is always online).

But building a simple RAG/Vector DB on top of llama3 is not very complicated.