- RAG generally gets you to prototype stage with interesting/demo-able results quickly, however, if your users turn out to submit queries that the embedder finds hard to vectorize (meaning you don't retrieve relevant vectorized chunks of source data) then it quickly becomes painful
- It's easy to go overboard with prompt chains that do summarizing/reducing of the fetched results, or pre-processing prompts that help vectorize the query better (see pain point above); always invest in some testing framework & sane test data upfront so you can avoid the classical data science "tweak it until the demo looks good" trap
- Don't ever use langchain, it's a baroque shitshow of a tool with a cluttered, inconsistent API written with a bad, inefficient coding style
- Paying for bespoke vector databases is probably snake oil and besides the weird pricing only causes pain in the long run when you want to store more than just your embeddings (looking at you 70$+ usd/month Pinecone); postgres with pgvector gets you very far until you hit multiple million documents, and you get all the benefits of a mature, scalable rdbms. Keep your embeddings close to the rest of your data.