Interesting performance with GPT 3.5, what does performance on Llama look like? What about smaller models like Llama 3.1?
16 karma · joined March 16, 2023
Probably the main point I disagree with you is that RAG is just ETL. If that was the case, all of the AI apps people are building would be AMAZING because we solved the ETL problem years ago. Yet, app after app being released have issues like hallucinations and incorrect data. IMO the second you insert a non-deterministic entity in the middle of an ETL pipeline, it is no longer just ETL. To try to add value here, our focus has been on adding capabilities to the framework around data synchronization (which is actually more of a vector management problem), contextualization of data through metadata and retrieval (this part being were we have spent the least time to date, but are currently spending the most)
There are a couple areas where we think we are driving some differentiation.
1. The management of metadata as a first class citizen. This includes capturing metadata at every stage of the pipeline.
2. Be infra ready. We are still evolving this point, but we want to add abstractions that can help developers apply this type of framework to a large scale distributed architecture.
3. Enable different types of data synchronization natively. So far we enable both full and delta syncs, but have work in the pipeline to bring in abstractions for real-time syncing. 3.
The idea of auditing the strategy is interesting. The flow that we have used for the semantic chunkers up to date has been along these lines where we : 1) Use the utility to generate the code snippets (and do some manual inspection) 2) Test the code snippets against some sample text 3) Validate the results
Generally we have found that recursive chunking and character chunking tend to be short sighted.
Today, it is mostly about convenience. We provide abstractions in the form of a pipeline that encompasses a data source, embed and sink definition. This means that you don't have to think about embedding your query or what class you used to add the data into the vector DB.
In the future, we have some additional abstractions that we are adding that will add more convenience. For example, we are working on a concept of pipeline collections so that you can search across multiple indexes but get unified results. We are also adding more automation around metadata given that as part of the pipeline configuration we know what metadata was added and examples of it, so we can help translate queries into hybrid search. I think about it as a self-query retriever from Langchain or Llama Index but that automatically has context of the data at hand. (no need to provide attributes)
Are there any specific retrieval capabilities you are looking for?
IMO the fun parts are actually prototyping and figuring out the right pattern I want to use for my solution. Once you have done that, scaling and dealing with robustness tends to be a bit less fun.
We do support updates for some sources. Deletes not yet. For some sources we do polling which is then dumped on the queues. For other we have listeners that subscribe to changes.
What are the challenges you are facing in supporting this?
Curious on what type of customization are you looking to add that you would want something like a library?
For synthetic data scenarios are you also thinking about synthetic queries over the data? (Try to predict which chunks might be more used than others)
Our current perspective has been on leveraging LLMs as part of async processes to help analyze data. This only really works when your data follows a template where I might be able to apply the analysis to a vast number of documents. Alternatively it becomes too expensive to do at a per document basis.
What types of analysis are you doing with LLMs? Have you started to integrate some of these into your existing solution?
Were you using postgres already or migrated data into it?
We have been experimenting with semantic chunking (https://www.neum.ai/post/contextually-splitting-documents) and semantic selectors (https://www.neum.ai/post/semantic-selectors-for-structured-d...) but from a scale perspective. For example, if we have 1 millions docs, but we know they are generally similar in format / template, then we can bypass having to use an LLM to analyze them one by one and simply help create scripts to extract the right info.
We think there are clever approaches like this that can help improve RAG while still being scalable.
Yeah a ton of the time and effort has gone into building robustness and observability into the process. When dealing with millions of files, a failure half way through it is imperative to be able to recover.
RE: Weaviate: Yeah, we needed to use large amounts of memory with Weaviate which has been a drawback from a cost perspective, but that from a performance perspective delivers on the requirements of our customers. (on Weaviate we explored using product quantization. )
What type of performance have you gotten with Lance both on ingestion and retieval? Is disk retrieval fast enough?
You are right. Retrieval accuracy is important as well. From an accuracy perspective, any tools you have found useful in helping validate retrieval accuracy?
In our current architecture, all the different pieces within the RAG ingestion pipeline are modifiable to be able to improve loading, chunking and embedding.
As part of our development process, we have started to enable other tools that we don't talk as much in the article about including a pre processing and embeddings playground (https://www.neum.ai/post/pre-processing-playground) to be able to test different combinations of modules against a piece of text. The idea being that you can establish you ideal pipeline / transformations that can then be scaled.
Recently, I have heard a lot of chatter about using LLMs on top of structured data. At Neum AI, we have been dealing with our fair share of this and wanted to create a small blog outlining some of our thinking and best practices. Also show developers that might be new to the space what are some of the basics on how to get started. Most of the blog focuses on the intersection of structured data and retrieval augmented generation as that is the area where we have seen most questions. I do acknowledge that there are other places where LLMs can intersect with structured data like using LLMs to help write queries, but I am sure at this point we have all seen at least one or two services that do that.
As part of the blog, I included a small code sample showcasing how to build a simple app that consumes CSV files and using Langchain + Chroma can be queried for information.