227 karma · joined July 14, 2019
We've just released R2R V3 with a completely RESTful API that covers everything you need for production RAG applications. The biggest change is our Git-like knowledge graph architecture, but we've also unified all the core objects you need to build real applications.
Check our [API](https://r2r-docs.sciphi.ai/api-and-sdks/introduction) or join our [Discord](https://discord.gg/p6KqD2kjtB) if you want to dive deeper. We'd love feedback from folks building in production!
The quickstart clearly has too much content in retrospect, and the feedback here makes it clear we should simplify.
Think of R2R as an SDK with an out of the box admin dashboard / playground that you can plug into.
The key advantages can be extracted from the response above to Kluless -
R2R is built around RESTful API and is dockerized, so devs can get started on app development immediately.
The system was designed so that devs can typically scale data ingestion up to provider bottlenecks w/out extra work.
We have implemented user-level permissions and high level document management alongside the vector db, which most devs need to build in a production setting, along with the API and data ingestion scaling.
Lastly, we also log every search and RAG completion that flows through the system. This is really important to find weaknesses and tune the system over time. Most devs end up needing an observability solution for their RAG.
All of these connect to an open source developer dashboard that allows you to see uploaded files, test different configs, etc.
These basic features mean that devs can spend more time on iterating / customizing their application specific features like custom data ingestion, hybrid search and advanced RAG.
For this we have a target dataset (the YC co directory) that we have around 100 questions over. We have found that when feeding an entire company listing in along with a single question we can get an accurate single answer (needle in haystack problem).
So to build our evaluation dataset we feed each question with each sample into the cheapest LLM we can find that reliably handles the job. We then aggregate the results.
This is not perfect but it allows us to have a way to benchmark our knowledge graph construction and querying strategy so that we can tune the system ourselves.
we have instructions for getting setup and running w/ ollama. It should be pretty smooth.
For instance, if a user asks "How does A compare to B" then the query expansion element of HyDE is incredibly useful. The actual value of translating queries into answers for embedding is a bit unclear, since most embedding models we are using have been ft'ed to map queries onto answers.
I'm just curious because we have theorized and seen in practice that extraction is a way to answer questions which require connected information across disparate chunks, like you can see in the simple cookbook here [https://r2r-docs.sciphi.ai/cookbooks/knowledge-graph].
Or do you think this is something that can just be solved with more advanced multimodal ingestion?
csv, docx, html, json, md, pdf, pptx, txt, xlsx, gif, jpg, png, svg, mp3, mp4.
There are a lot of good questions around ingestion today, so we will likely figure out how to intelligently expand this.
For mp3s we use whisper to transcribe, for videos we transcribe with whisper and sample frames to "describe" with a multimodal model. For images we again transcribe to a thorough text description - https://r2r-docs.sciphi.ai/cookbooks/multimodal
We have been testing multi-modal embedding models and open source models to do the description generation. If anyone has suggestions on SOTA techniques that work well at scale we would love to chat and work to implement these. Long run we'd like the system to be able to handle multi-modal data locally.
In the meantime we've built our own dashboard which shows ingested documents, and has a customizeable chat interface - https://github.com/SciPhi-AI/R2R-Dashboard.
It's still a bit rough though.
We've talked to a lot of different developers about these problems and haven't found a general consensus on what features are needed, so we are still evaluating advanced approaches.
For now our implementation is more general and designed to work across a variety of documents. R2R was designed to be very easy to override with your own custom parsing logic for these reasons.
Lastly, we have been focusing a lot of our effort on knowledge graphs since they provide an alternative way to enhance RAG systems. We are training our own model for triples extraction that will combine with the automatic knowledge graph construction in R2R. We are planning to release this in the coming weeks and are currently looking for beta testers [we made a signup form, here - https://forms.gle/g9x3pLpqx2kCPmcg6 for anyone interested]
Since multiple people have requested we are pushing a quick change to make this emphasized in the docs.
To do what you are requesting is pretty easy, you can just launch the server and use the client directly. The code would look like this:
```python
from r2r import R2RClient
base_url = "http://localhost:8000" # or other
client = R2RClient(base_url)
# load my_file_paths
...
response = client.ingest_files(file_paths=my_file_paths)
# optionally set metadata, document ids, etc.. [https://r2r-docs.sciphi.ai/api-reference/endpoint/ingest_fil...]
```
Ingestion is pretty straightforward, you can call R2R directly or use the client-server interface to pass the html files in directly to the ingest_files endpoint (https://r2r-docs.sciphi.ai/api-reference/endpoint/ingest_fil...).
The data parsers are all fairly simple and easy to customize. Right now we use bs4 for handling HTML but have been considering other approaches.
What specific features around ingestion have you found lacking?
R2R is built around RESTful API and is dockerized, so devs can get started on app development immediately.
The system was designed so that devs can typically scale data ingestion up to provider bottlenecks w/out extra work.
We have implemented user-level permissions and high level document management alongside the vector db, which most devs need to build in a production setting, along with the API and data ingestion scaling.
Lastly, we also log every search and RAG completion that flows through the system. This is really important to find weaknesses and tune the system over time. Most devs end up needing an observability solution for their RAG.
All of these connect to an open source developer dashboard that allows you to see uploaded files, test different configs, etc.
These basic features mean that devs can spend more time on iterating / customizing their application specific features like custom data ingestion, hybrid search and advanced RAG.
You can see the client here - [https://github.com/SciPhi-AI/R2R/blob/main/r2r/client/base.p...].
Do you have any preferred frameworks?
This is when the relational nature of postgres really shines - you can even do in-mem calculations for distance.
The infra challenges are real - it has been what I have been struggling the most with in providing high quality support for early users. Most want to be able to reliably firehose 10-100s of GBs of data through a brittle multistep pipeline. This was something I struggled with when building AgentSearch [https://huggingface.co/datasets/SciPhi/AgentSearch-V1] with LOCAL data - so introducing the networking component only makes things that much harder.
I think we have a lot of work to do to robustly solve this problem, but I'm confident that there is an opportunity to build a framework that results in net positives for the developer.
FWIW, Your feedback would be invaluable as the project continues to grow.