jamesbriggs

We used frameworks in the past, tried langchain, langgraph, and Openai's agents SDK pretty extensively. Now we roll our own, generally a much better and cleaner experience. We essentially built our own internal framework for our own use-case, we liked the graph approach of langgraph - so we took elements of that. We write everything async, and added nice handling for streaming.

You can see our framework [here](https://github.com/aurelio-labs/graphai). I don't necessarily recommend it as it's built for our use-cases and I make no guarantees for others, but it might be interesting to see what rolling your own might look like

My bad, the link was wrong - you found the right one. I've updated it in the repo too, thanks. Let me know how it goes!

Sharing my walkthrough on fine-tuning LLMs with LoRA using NVIDIA's NeMo microservices. The result is a llama-3.2-1b-instruct model fine-tuned to be really good at function-calling, making it ideal for agent-use.

It was a ton of fun to figure it out and it brought back some nostalgia from the days of training ML models, tweaking learning rates, dropout, and watching loss charts in W&B.

Final performance was way better than any 1-3B parameter LLM I tried with agentic workflows in the past.

Releasing this walkthrough on fine-tuning LLMs with LoRA using NVIDIA's NeMo Microservices (they sponsored the video, but with no reqs on what I do or say). We cover a ton on building prod AI applications, including:

- LoRA fine-tuning of Llama 3.2 1B - Deploy of private AI agent systems - Using NVIDIA NIMs to host our fine-tuned LLM - Interacting with our LLM + streamed output

It was a ton of fun to figure this out, and it brought back some nostalgia from the good old days of training ML models, tweaking learning rates and dropout, and watching loss charts in W&B.

The result is a llama-3.2-1b-instruct fine-tuned to provide pretty good function-calling abilities (better than any other out-of-the-box 1-3B models I tried).

I made a course covering everything you'd need to know to start building with LangChain. It assumes no level of expertise, you can be a complete beginner and by the end of the course be building AI agents with chat memory, streaming, async, etc. There's no cost, it's all free!

Tutorial showing how to do faster and more deterministic function calling using the open source Semantic Router library

I and my team have been building a python library for improving the steerability of AI agents, the library also allows us to add an essentially unlimited number of tools to agents, add safety guardrails, etc. We have a lot more coming and I'm just excited to share our first public release. Let us know your thoughts!

Not weighing on whether it is AGI-like or not, but my view on the search component is that it's like when we google info, we have a search term and we don't know whether that will return good results or not. So I suppose in order for RAG to become better the LLM needs to be able to review the information given, and decide whether to search more (which they can do with agents to be fair), and we also need to be building retrieval systems that do very well at returning the right information even with sub-optimal search queries.

I think LLMs are fairly good at deciding what information is relevant to a query or not though, it is rare that I have found an LLM to be distracted by RAG results (even when they're not relevant)

Yeah RAG has been around for some time, paper [2] being where I first stumbled onto it — I remember a 2-3 years ago building a RAG pipeline using either the model they trained in paper 2, or that was trained based on the same idea, with Deepset AI's Haystack library. It has been a little surreal to see that terminology become so popular again

I may be misunderstanding, but I'll try to answer — quantization typically means retrieval will be slower (if referring to techniques like product quantization), but that is the case whether you're at 10K vectors or 1B vectors, afaik it doesn't really make a difference because you're only quantizing the query vector at query time (it has been awhile since I read anything on quantization, so I could be mistaken).

Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index)

So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.

Very cool, how are you doing the speech-to-text part, with Whisper?

For Huberman Podcast I imagine he pays someone to do the annotations manually, so they're accurate. But on most videos I've found Whisper's annotations to be more accurate than YouTube's default annotations - not to bash YouTube's, they're still great but occasionally you get some weird annotations

I should add, Riley used the ada embedding model (rather than sentence transformers). Performance wise they should be similar (in ability to encode meaning accurately) but the ada model can encode a much larger chunk of text. I don't know exact numbers but something like 1-2 pages of text in a typical corporate PDF. Whereas sentence transformers are typically limited to around a paragraph of text.

Typically you'd split the text in paragraph sized chunks to handle this requirement of sentence transformers, with GPT-3 embeddings you naturally have more flexibility there

you can return the chunks of text containing the answers, but not generate answers as that isn't what text-embedding-ada-002 is for. For that you need generation model (davinci in this case)

I built something similar using a variety of YouTube channels focused on NLP, AI, etc. The app is here https://huggingface.co/spaces/jamescalam/ask-youtube - you can ask things like "what is a transformer model?" or "what is semantic search?"

The way I built it is documented here: https://www.pinecone.io/learn/openai-whisper/

Afaik it's the same approach as Riley, that is:

- Scrape audio of YouTube videos

- Transcribe to text with OpenAI's Whisper

- Use sentence transformer to create embeddings of text

- Index embeddings (with transcribed text, timestamps, and video URL attached) in Pinecone's vector database

- Wrap up the querying functionality in a nice UI

(this is for the search functionality)

If wanting to replicate the Q&A part, I also built something similar and wrote about it (https://youtu.be/coaaSxys5so) - it's essentially the same process but we return text snippets to GPT-3 along with the original question and it generates an answer

Hi, author of the article here.

They act as a "position signal" that modifies the patch embedding. The learned signals are similar to other neighbouring position signals, and the later layers of the model will use the "similarity" between signals to identify the proximity/order of different patches.

There is no explicit mechanism that tells the network to make neighboring position embeddings similar, it's just a result of the training that fortunately works and seems logical.

I can definitely try to explain further if needed

Intro to a series of content I'm producing on Hugging Face datasets, one of the most coolest libraries for ML

Faiss is great, but I see there being several strong reasons for using Pinecone:

1. The people who built the Pinecone vector index include some of the top experts in information retrieval/vector search in the world, the chances of someone like myself building something more efficient in Faiss is pretty slim, even after spending months learning the library and as much as I can about vector search

2. Faiss is only a vector index, it doesn't manage your data, so you need to personally build a framework around your Faiss index to handle that, speaking from experience this can be frustrating

3. Pinecone has one of (if not the) most advanced implementation of filtering available, they use 'single stage' filtering, which provides pre-filtering speeds with post-filtering accuracy

4. Scalability, want to search through an index of 2B vectors? Not an issue, getting that working with Faiss is not fun

5. Ease of use, all of the above takes an API key and ~5 lines of code to setup

I think the use case of chatbots is better solved with open domain Q&A (eg https://www.pinecone.io/learn/question-answering/). The focus of most chatbots seems to be on answering questions, but wrapping it up in a nice interface. That's fine but the chatbot can only answer (accurately) questions that have an answer somewhere (probably buried deep in some Q&A pages). It's much more user-friendly imo to have a Google type interface where you can answer a question, and return answers, or at least get an idea of where the answer is - and open domain Q&A does this fairly well (as proven by Google)

You can train a good sentence transformer on ~10K sentences using TSDAE (an unsupervised training approach), covered in Chapter 7 here: https://www.pinecone.io/learn/unsupervised-training-sentence...

Agree that this would be super helpful

The HuggingFace course is very good for NLP: https://huggingface.co/course/chapter1/1

I don't remember the paper, but the authors working on this same problem suggested to do this + also include an overlap between spans.

It will depend on your use-case, but primarily:

(1) Pinecone uses dense vectors which can encode much more meaningful info, eg the actual 'semantic meaning' behind a sentence as we (people) would understand it, or the context in an image. Because of this, we can enable much richer, human-like interaction/search in your applications

(2) Performance wise, before joining Pinecone I was spending a lot of time with other dense vectors search tools like Faiss, and it isn't easy to get good or even reasonable accuracy and latency, particularly for large datasets. When I first used Pinecone, it took me maybe 10 minutes to figure everything out and start querying a reasonable dataset, search times were very fast and the accuracy incredible. Pinecone's tech is built by people that live and breath vector search, and what they've built outperforms anything I can build, even if I spend months trying to build it. I got better performance with Pinecone in 10 mins.

(3) Everything is production ready, no need to worry about deployment, security, maintenance etc, Pinecone deal with it and you can even use the service for free up to 1M vectors.

Ideally I think it's useful to be familiar with Python and some NLP. Parts of the course can get reasonably in-depth, but we have tried to simplify every as much as possible and I think much of it can be followed without too much prerequisite knowledge.

That being said, as a writer it sounds like you'd be most interested in generative models that generate new text, we do cover that a little in the Question Answering chapter, but otherwise, not so much. Although I don't know your ideas or plans so I can't say for sure what is good/bad for you to focus on.

I do think you should read into transformer models like BERT and GPT, there is a very good free course from HuggingFace who are the 'de-facto framework' for NLP at the moment, that could be useful for you too: https://huggingface.co/course/chapter1/1 (chapter one will introduce everything too, definitely go through that)

With HuggingFace, you can put together a simple T5 or GPT-2 text generation script with a few lines of code

That's awesome, glad to hear it!

Thanks!

Hi all,

I'm James, past data scientist, ML engineer, and now dev advocate at Pinecone.

I've been dedicating much of the past few months to building a course on what I've found to be one of the most fascinating emerging technologies in AI and ML, Semantic Search:

https://www.pinecone.io/learn/nlp

The course is completely free, it includes written and visual explanations, code implementations, and video walkthroughs. It focuses on the NLP side of semantic search and the first seven chapters cover:

1. Dense Vectors

2. Sentence Embeddings and Transformers

3. Training Sentence Transformers with Softmax Loss

4. Training Sentence Transformers with MNR Loss

5. Multilingual Sentence Transformers

6. Question Answering

7. Unsupervised Training for Sentence Transformers

There is more coming soon and I'll be around to answer any questions, let me know what you think.

Thanks all!

Voronoi is super popular and typically is a component of solutions used in practice, we did write another article that covers what some of those might look like: https://www.pinecone.io/learn/composite-indexes/

TLDR Voronoi + HNSW is popular

I think Haystack is awesome but when the Faiss vector index starts getting too large their default 'Flat' index is too slow. I think a lot of people miss that they can fix that using custom Faiss indexes, which are set using `faiss_index_factory_str` when initializing the index.

I built a big course on vector search (it's free), and more specific to this problem, an article covering these index factory strings and how to build good indexes with them - hopefully it's useful!

https://www.pinecone.io/learn/composite-indexes/