But the later transformer models have a good grasp of grammar - similar to when we learn to speak as children, it isn't necessary to be taught grammar directly, but we can figure it out over time based on the patterns of language.
193 karma · joined June 29, 2021
But the later transformer models have a good grasp of grammar - similar to when we learn to speak as children, it isn't necessary to be taught grammar directly, but we can figure it out over time based on the patterns of language.
Google do this and they're very good at it, but a lot of companies need their own search capabilities - think about those internal help pages. They usually seem super outdated compared to the semantic search capabilities of Google.
In the end there's only a few components to it, you use some NLP model to create what are called 'dense vectors'. Then you put all these dense vectors into an 'index' which is optimized for fast search (that comes under the umbrella of ANN search). Then given a new query you just compare that to the items in the index and return the most similar results.
I covered the search part of it (https://www.pinecone.io/learn/) with Pinecone, who provide managed-search - although here we mainly focus on Faiss (a great engine from Facebook AI). However, we're also looking to create some content covering the first half too, which is 'how to build dense vectors' using models like BERT, we have one post so far on that (https://www.pinecone.io/learn/dense-vector-embeddings-nlp/)
It producing interesting results, but doesn't really progress the field - although who knows, maybe skynet is actually a 100T parameter transformer
I came across this paper when researching for the article, I agree the link to brain networks is fascinating
* IVFPQ seems reasonable but unlikely to hit 90%
* HNSW could be a fast and accurate alternative
* a Flat IVF index could definitely reach 90%, but it may be slow