Comment by simonw | Hacker News Reader

Comment by simonw | Hacker News Reader

The challenge with this kind of system is always the bit that figures out the most relevant text from the corpus to bake together into the prompt.

It's interesting to see that this example takes the simplest approach possible, and it seems to provide pretty decent results:

> Third, each word of the list cleaned up above is searched inside the information paragraph. When a word is found, the whole sentence that includes it is extracted. All the sentences found for each and all of the relevant words are put together into a paragraph that is then fed to GPT-3 for few-shot learning.

This is stripping punctuation and stopwords and then doing a straight string match to find the relevant sentences to include in the prompt!

A lot of people - myself included - have been trying semantic search using embeddings to solve this. I wrote about my approach here: https://simonwillison.net/2023/Jan/13/semantic-search-answer...

My version dumps in entire truncated blog entries, but from this piece I'm thinking that breaking down to much smaller snippets (maybe even at the sentence level) is worth investigating further.

One thing that seems like it may be a slight issue for me is that some questions don't actually need any knowledgebase information, and when I include the closest matches it can confuse text-davinci-003. So I added something like "ignore if none of these snippets are relevant" but still had an issue so I ended making a separate command to turn on the kb search per question.

I'm wondering if there is some cosine similarity cut off I could use to just drop kb matches, but it seems like probably not because a lot of real matches are pretty close in similarity to non-matches.

I see your blog referencing what you call the “semantic search answers” pattern. I’ve seen this elsewhere as Retrieval Augmented Generation or Data Augmented Generation. A few libraries, like LangChain, have support for this.

Thanks, I just updated the blog entry to add a link to the Retrieval Augmented Generation paper.

I’ve been playing around with this and have gotten decent results simply using openAI embeddings. I’ll probably be making some kind of post showing results soon

Yeah I'm happy with the results I got from embeddings so far. The areas I want to explore there are:

1. What's the ideal size of text to embed? I'm doing whole blog entries right now but I'm confident I can get better results if I divide them up into smaller chunks first - I'm just not sure how best to do that.

2. There's a trick called Hypothetical Document Embeddings (HyDE) where you ask GPT-3 to invent an answer to the user's question, embed THAT fictional answer, then use that embedding to find relevant documents in your corpus. https://arxiv.org/abs/2212.10496

Wow thanks for sharing that’s such a clever hack

For 1. I think you may find work with semantic text chunking interesting, and the possibility of overlaying embeddings with one another (e.g. powering fragen.co.uk youtube search, the transcriptions whenever you click the "play" icon are semantically chunked...). Audacity has a good Autochapter API for this too. I found after good OCR postprocessing, semantic splitting is most important.

For 2 - have you used this or found it to work? I found asymmetric embeddings like MS MARCO trained models much superior when you have a reasonably bespoke corpus (like the demos of yours I've seen) - because, TLDR, it had a negative impact on recall, even as it improved Precision - if your GPT hypothetical answer is a misconception or joke answer, you'll be semantic searching joke answers, etc - same with numbers, GPT-3 might suggest short or plurality-default answer, like US phone numbers for "what is head office of marketings number?" when you have UK numbers in your corpus you would prefer to be surfaced.

I’ve done some experiments and the ideal seems to be semantic chunks of max 175 tokens. But only if those chunks are already very dense (use summarisation to get there)

Reply on news.ycombinator.com