HNHacker News
TopNewBestAskShowJobs

jamesbriggs

193 karma · joined June 29, 2021

submissionscomments
jamesbriggs··on Dense Vectors: Capturing Meaning with Code
These models capture similarity on such a scale that they manage to learn patterns of language well, in terms of grammar they learn it indirectly - it is not manually coded, particularly the later models that we speak about. Our focus here is more on encoding text (and images) in a way that they can be compared to other text or image based on their meaning/imagery rather than syntax

But the later transformer models have a good grasp of grammar - similar to when we learn to speak as children, it isn't necessary to be taught grammar directly, but we can figure it out over time based on the patterns of language.

jamesbriggs··on What software engineers should know about search (2017)
A lot of websites scrape Medium without the authors having any idea, I assume it is another one of those
jamesbriggs··on What software engineers should know about search (2017)
Being from 2017 the article misses some of the coolest advances in semantic search, which is now pretty easy and lets you search in (almost) the same way they would ask a shop assistant when looking for something specific - "do you know where the thing with the cool circles and pointy bits is?" (maybe being a little more specific...)

Google do this and they're very good at it, but a lot of companies need their own search capabilities - think about those internal help pages. They usually seem super outdated compared to the semantic search capabilities of Google.

In the end there's only a few components to it, you use some NLP model to create what are called 'dense vectors'. Then you put all these dense vectors into an 'index' which is optimized for fast search (that comes under the umbrella of ANN search). Then given a new query you just compare that to the items in the index and return the most similar results.

I covered the search part of it (https://www.pinecone.io/learn/) with Pinecone, who provide managed-search - although here we mainly focus on Faiss (a great engine from Facebook AI). However, we're also looking to create some content covering the first half too, which is 'how to build dense vectors' using models like BERT, we have one post so far on that (https://www.pinecone.io/learn/dense-vector-embeddings-nlp/)

jamesbriggs··on Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
Also agree with this, it's almost like a marketing ploy - especially from OpenAI. They produce awesome stuff but things with GPT can get silly sometimes, like when they wouldn't release the larger versions because 'they were too powerful', and in the end you ask it how many eyes my foot has and it says seven...

It producing interesting results, but doesn't really progress the field - although who knows, maybe skynet is actually a 100T parameter transformer

jamesbriggs··on Hierarchical Navigable Small Worlds
You also have plenty of comments which makes it much easier to follow than the Faiss code - I wish I had found this a few weeks ago, thanks for sharing.
jamesbriggs··on Hierarchical Navigable Small Worlds
Yes your explanation is correct, if it hits an end block or overshoot the target it returns to the previous block in the next layer (not the start block), I submitted a change to the article and visual to make this clear - should be updated soon. Thanks for pointing it out!
jamesbriggs··on Hierarchical Navigable Small Worlds
Yes I missed that, thanks for pointing it out!
jamesbriggs··on Hierarchical Navigable Small Worlds
Yes that's why we wrote it! Other than in papers, finding info on HNSW is near-impossible so hopefully this will make a few lives easier.

I came across this paper when researching for the article, I agree the link to brain networks is fascinating

jamesbriggs··on Hierarchical Navigable Small Worlds
Hey I'm the author, feel free to ask me any questions!
jamesbriggs··on The Missing ”WHERE” Clause for Vector Search
Yes as gk1 said, we're figuring out what can be said! I can't fully answer your questions beyond, having both indexes together helps us speed up the search (thanks to a lack of network/serialization/parsing overhead), and because of the structure we store metadata tags in, we also don't need to check the metadata for every sample (hence the cool speedup you get when filtering) - sorry that we can't say anything more yet!
jamesbriggs··on The Missing ”WHERE” Clause for Vector Search
Hi all, I'm the author - I'll be here to answer any questions you have!
jamesbriggs··on The Missing ”WHERE” Clause for Vector Search
Excalidraw.com for the style/graphs, then Adobe animate for the animation!
jamesbriggs··on Product Quantization: Compressing high-dimensional vectors
Given the top 100 results produced by a flat index - how many of those are returned in the top 100 of our PQ/IVFPQ index. Although it's worth noting that this is at the lower end of performance for PQ, using the same Sift1M dataset we can hit 76% with IVFPQ using nlist=256, m=32, and nbits=8. I'm sure this could be increased with a little more fine-tuning too. A couple of options:

* IVFPQ seems reasonable but unlikely to hit 90%

* HNSW could be a fast and accurate alternative

* a Flat IVF index could definitely reach 90%, but it may be slow

jamesbriggs··on Product Quantization: Compressing high-dimensional vectors
Excalidraw + Adobe Animate - I have never seen Excalidraw-animate before though so this could be new option!
jamesbriggs··on Nearest neighbor indexes for similarity search
Welcome! Unfortunately faiss doesn't support sparse vectors. If you're using sparse vectors I would either not use faiss (there are better options like Elasticsearch) - or convert them into dense for use with faiss
jamesbriggs··on Nearest neighbor indexes for similarity search
Hey all, I'm the author - feel free to ask me any questions about the article!
← PreviousPage 2 of 2