kaycebasques

Apparently, Musk is very popular among Republicans: https://www.politico.com/news/magazine/2025/06/06/elon-musk-...

Thank you for the details. I was thinking about ML embeddings in particular, but your answer does help me understand the core mathematical nature of an embedding at a deeper level. E.g. the word "embedding" makes more sense now in the context of "one mathematical structure stuck inside another."

The post title reminds me of something that I researched a little a couple months back. Practically all embeddings are implemented as vectors, right? Definitionally, an embedding doesn't have to be a vector. But in practice there's not really any such thing as a non-vector embedding, is there?

One thing I learned recently is that, if your embedding model supports task types (clustering, STS, retrieval, etc.), then that can have a non-trivial impact on the generated embedding for a given text: https://technicalwriting.dev/ml/embeddings/tasks/index.html

Parquet and Polars sound very promising for reducing embeddings storage requirements. Still haven't tinkered with them: https://minimaxir.com/2025/02/embeddings-parquet/

And this post gave me a lot more awareness to be more careful about how exactly I'm comparing embeddings. OP's post seems to do a good job explaining common techniques, too. https://p.migdal.pl/blog/2025/01/dont-use-cosine-similarity/

In terms of "best album when listening all the way through, from start to finish", De-Loused is my number one. The way that they have these phenomenally catchy choruses, and then go off on captivating jams or ambient sound, and then come back to those choruses... chills. One of my big regrets is not seeing that album tour live.

Second album was also very enjoyable for me.

So, based on the emphasis on "retrieval" in the blog title, are these models narrowly focused on retrieval tasks only? E.g. I shouldn't use them for clustering, STS, and so on?

How does voyage-3.5 compare against Gemini Embedding (GE)? I thought GE had top spot for retrieval tasks on MMTEB. Is Voyage saying here that voyage-3.5 now has the top spot? Or is it just that, for the 10 specified datasets, voyage-3.5 outperforms the specified OpenAI and Cohere models.

A common verb "now"??

> Grok (/ˈɡrɒk/) is a neologism coined by the American writer Robert A. Heinlein for his 1961 science fiction novel Stranger in a Strange Land. While the Oxford English Dictionary summarizes the meaning of grok as "to understand intuitively or by empathy, to establish rapport with" and "to empathize or communicate sympathetically (with); also, to experience enjoyment",[1] Heinlein's concept is far more nuanced, with critic Istvan Csicsery-Ronay Jr. observing that "the book's major theme can be seen as an extended definition of the term."[2] The concept of grok garnered significant critical scrutiny in the years after the book's initial publication. The term and aspects of the underlying concept have become part of communities such as computer science.

https://en.wikipedia.org/wiki/Grok

> The idea behind the search engine is to embed each of my posts into this domain by adding up the embeddings for the words in the post.

Ah, OK! I never really grokked how to use word-level embeddings. Makes more sense now.

Effective error messages are one of my platonic ideals about how documentation is supposed to work. The docs shouldn't be shoved off to the side. They should appear just-in-time, when you need them. If you can fix the error within the product (e.g. when Rust tells you exactly how to fix the typo), just do it there. Otherwise link off to the full docs when it's too much content.

The general platonic ideal is "have the product automatically fix the issue" => "provide short documentation within the product if the problem can be explained with just a paragraph or two of content" => "link to a targeted doc that deals with this exact problem if it takes more than a few paragraphs to explain"

A lot of time, my work as a technical writer is advocating to update the product (or updating the product myself) to do the first two steps, rather than just jumping immediately to the last step. Startup people often refer to this as "the perfect product requires 0 documentation." When teams always resort to fixing product issues with docs, your docs start to get huge and complicated. We technical writers often refer to this as "putting docs lipstick on the product pig."

(Question for anyone) how could I go about replicating this with Gemini Embedding? Generate and store an embedding for every word in the dictionary?

(Tangential) On a recent roadtrip up to Portland from SF I stopped in a small historic mining town near Shasta called Dunsmuir. They had a Little Free Library so of course I had to check out what was in it. I was delighted to find an old book on Forth from the 80s, called Starting Forth. Inside of the book there were some business cards for FIG: Silicon Valley Forth Interest Group.

Will update the post to capture that interesting tidbit, thanks

And also, thanks for this post! https://minimaxir.com/2025/02/embeddings-parquet/

Yes, I have experimented with using ML for style concerns such as consistent capitalization and verbosity and can confirm: no need for embeddings here. You can get the job done with only a text generation model. I actually can't visualize how an embeddings-powered implementation would work here

IMO, for technical writing, citing a page or section within a page is usually good enough. I rarely need to cite a particular concept. But I've never even thought of the possibility of more granular concept-level citations and will definitely be pondering it more!

For word2vec I know that there's a bunch of demos that let you do the king - man + woman computation, but I don't know how you do this with modern embeddings. https://turbomaze.github.io/word2vecjson/

Yes, confirmed here: https://news.ycombinator.com/item?id=43966937

Follow-up question based on your semantic algebra idea. If you can start with an embedding and generate semantically similar text, does that mean that "length of text" is also one of the properties that embeddings capture?

87MB is still quite big, though. Think of all the comments here on HN where people were appalled at a certain site loading 10-50 MB of images. Hopefully browser vendors will figure out a secure way to download a model once and re-use that single model on any website that requests it. Rather than potentially downloading a separate instance of all-MiniLM-L6-v2 for each site. I know that Chrome has an AI initiative but I didn't see any docs about this particular problem: https://developer.chrome.com/docs/ai

Cool. I kinda grok the idea of semantic scrolling but I'm having trouble seeing it in action in the site. I think it would be useful in cases where I want to become an expert in a given topic and therefore I want to peruse lots of related ideas and create the possibility of serendipitous new neural connections. As for technical documentation, usually people want to find certain information as quickly as possible so that they can get on with their work, so I don't think semantic scrolling would be a good fit on most docs sites. I.e. they won't have the patience to semantically scroll in order to find the info they need.

In-site search is super important. I suspect that many docs maintainers don't realize how heavily it's used. Many docs sites don't even track in-site search queries!

One of the things I love about Sphinx is that it has a decent, client-side, JS-powered offline search. I recently hacked together a workflow for making it search-as-you-type [1]. jasonjmcghee's comment [2] has got me pondering whether we can augment it with transformer.js embeddings.

[1] https://github.com/orgs/sphinx-doc/discussions/13222

[2] https://news.ycombinator.com/item?id=43964913

Also, re: direct applications of embeddings in technical writing, see https://www.tdcommons.org/dpubs_series/8057/

Also re: a direct application I forgot to mention this: https://www.tdcommons.org/dpubs_series/8057/

(It finally published last week after being in review purgatory for months)

Thanks for the feedback. I've been reading the research papers behind how these models are created e.g. Gecko [1] and NV-Embed [2] and Gemini Embedding [3] so I'm starting to grok what you're getting at and will find a way to make the post more accurate on this front

[1] https://arxiv.org/abs/2403.20327

[2] https://arxiv.org/abs/2405.17428

[3] https://arxiv.org/abs/2503.07891

Wasn't on my radar! Thanks for the pointer. I'll look into it. It's hard to keep up with all the embedding models out there.

You're not alone. I've gotten lots of feedback that the post buries the lede [1] and is ineffective because it doesn't provide more concrete examples. I will fix it eventually, I promise! See here for context: https://news.ycombinator.com/item?id=43964584

[1] It's perhaps not even appropriate to say "buried the lede" because that implies the lede is dug back up at some point, whereas this post buries the lede and then forgets where the lede was buried!!

> I guess "good" technical writing no longer includes a thesis statement.

Thesis is outlined in the second paragraph:

> What embeddings offer to technical writers is the ability to discover connections between texts at previously impossible scales.

I think it's fair, however, to say that this post is ineffective because it does not provide concrete examples of the thesis in action. My only excuse is that I never intended for this to be a standalone post but life got in the way (in the best possible way!) https://news.ycombinator.com/item?id=43964584

> why would this be useful for technical writing?

You're not going to like this answer, because it's also vague. There are 3 intractable challenges in technical writing. Embeddings can help us make progress on all 3: https://technicalwriting.dev/strategy/challenges.html

See also https://www.tdcommons.org/dpubs_series/8057/

Oh cool, client-side JS-powered embeddings were not on my radar. That opens up a lot of applications for docs sites. Thanks for sharing.

Parquet and Polars are definitely on my radar, though, after reading this: https://minimaxir.com/2025/02/embeddings-parquet/

Yes, definitely need to follow through on the follow-up posts and projects that show exactly how we apply embeddings to TW. Examples (in all their forms) are truly magical in how effective they are as a teaching aid.

Yes, as a standalone post I can totally see how this is not persuasive because it's too vague and doesn't spell out specific applications. My only excuse is that I never intended this to be a standalone post; it was intended to be a conceptual primer supplemented by follow-up posts and projects exploring different applications of embeddings in technical writing. Hopefully the renewed attention on this post will motivate me to follow through on the follow-up content ;)

By results I assume that you're asking about the related pages experiment? The results were definitely promising. A lot of the calculated related pages were totally reasonable. E.g. if I'm reading https://www.sphinx-doc.org/en/master/development/html_themes... then it's very reasonable to assume that I may also be interested in https://www.sphinx-doc.org/en/master/usage/theming.html

I was worried that introducing the formal concept too quickly would feel a bit overwhelming for my fellow technical writers who are learning about embeddings for the first time, but I know that it's also annoying when a post makes you wait too long to get an answer to a question. So I'll find a way to provide a quick answer upfront. Thanks for the feedback.