Pyversity with Thomas van Dongen (Springer Nature)
Thomas is the creator of Pyversity, a fast, lightweight open-source python library for diversifying retrieval results!
Diversity is such an underrated topic in AI and Vector Databases. Whether searching through e-Commerce products or Scientific papers, we often want serendipity from our search engine, results that we were not expecting to find!
For example, say you ask "Who has accomplish the most in professional sports?". A relevance optimized search system might return 10 results all about Michale Jordan... whereas a diversity enhanced system would produce information about Michael Jordan, Tom Brady, Tiger Woods, ...
Rather than just return the relevance ranked search results, Pyversity uses methods such as Maximal Marginal Relevnace (MMR) or Determinantal Point Process (DPP) to achieve diverse results.
I learned a lot from this conversation exploring the general topic of diversity in vector spaces, diversification strategies from Maximal Marginal Relevance (MMR) to Determinantal point process (DPP), and more, as well as Thomas' work and thoughts on AI in Scientific Literature!
I hope you find it interesting!
YouTube: https://www.youtube.com/watch?v=L2N1qvfP7tg
Spotify: https://spotifycreators-web.app.link/e/FBaEZ9b5UYb