LexGrep Grep through all of Lex Fridman podcasts

- I first started with whisper to generate the transcript but eventually found a dump online where someone did the same thing :V (first step done)

- Next I tried speaker diarisation which is really easy to do in this scenario (as there are mostly 2 speakers at any time) and I could see Lex saying "welcome to my podcast" in almost all of his podcasts which made annotating the transcript easier.

- After this, its the usual create mini embeddings from a single podcast (I used OpenAI's text-embedding-ada-002) to generate the embeddings. I had to model the text in a specific way "This is a transcript between Lex Fridman and John Carmack.." (this adds the context for the conversation and helps powers query like: What did John Carmack think about {blah}) as the embeddings from the podcast with John Carmack will be closer than say some other podcast.

- Pinecone DB provided a free tier for doing cosine similarity of the generated embeddings which worked out perfectly for my use-case and I could generate with a high enough accuracy the related embeddings from the transcript.

- The final piece here is using davinci-text-003 which takes in the transcript and context and after a bit of prompt engineering (more like getting the LLM to give me the right thing and also say "no" when it can't), I have a working end to end search.

Its really exciting how this all comes together and the APIs allowed me (someone who has been a systems engineer for so so long) try out Machine Learning and quickly see it being beneficial and also fun.