A key innovation of search engines in the early Internet era was sketching and near duplicate detection (NDD). The top results from engines like AltaVista were often flooded with duplicate pages with tiny differences. NDD was crucial for search quality, and sketches are central to many other big data techniques, such as locality sensitive hashing.
We’ve found that it’s just as relevant today for RAG as enterprises scale document collections and other generative AI use cases.