Comment by godelski | Hacker News Reader

Comment by godelski | Hacker News Reader

Parent Full thread

I'm a bit confused by the comment because it would seem that these are all relatively common tasks. The first and third being identical. Good example is in vision you might want to semantically change the photo like adding a pair of glasses or doing one of those things you see on the Google commercials. That's done in a latent space

I think this is clearest in the Normalizing Flow cases because you're just turning your space into a Gaussian (diffusion does this too, but through approximation methods and isn't invertible, though it is reversible). You project the image/sentence/data you want to manipulate, manipulate within gaussian space, then return back to target space.

Or maybe my confusion is shared confusion because embedding is an overloaded word that means a lot of things? Maybe you're just thinking of the first block that converts discrete integer tokens into continuous floats? But we learn those embeddings so even though it becomes like a lookup table it's still a neural process. People do things like SVMs on this space alone. But I think it is like latent space, which is only a bit more abstract. At least embeddings need to be injective, well... mathematically...

> relatively common

I'd love to see some links, all I see used in practice (including in the OP blog post) is semantic search and a bit of clustering

> adding a pair of glasses

Actually that (and generally all the SD/VAE stuff) is a great example of the kind of thing I was thinking of, though I have yet to see that concept being used together with a vector database; generally all the user-facing stuff I've seen fits it into the standard "train a model, then do inference" workflow, in contrast to something like semantic search which more obviously focuses on the embeddings themselves

> First and third being identical

Definitely related, but I make the distinction between projection/sorting along an axis vs constructing a new vector by addition/subtraction

> Manipulate within gaussian space, then return to target space

This is definitely along the lines of what I had in mind, any example of this being used in practice?

> Embedding is an overloaded word

Yeah I'm using the term somewhat loosely and broadly here, as basically "a vector in a real vector space where distance represents some notion of semantic similarity"

> People do things like SVMs

Who?

Here's an example from a Normalizing Flow. Good at density, not great at sampling.

https://openai.com/research/glow

Here's a video of moving around in the latent space of a diffusion model

https://www.youtube.com/watch?v=vEnetcj_728

Here's a stylegan one

https://www.youtube.com/watch?v=bRrS74RXsSM

Or a VAE on mnist

https://www.youtube.com/shorts/pgmnCU_DxzM

I mean it is a bit hard to answer your other questions because like I was pointing out, embeddings and latent spaces are pretty vague terms. For the mathy side, normalizing flows are a great choice since you can parameterize whatever data you want into whatever distribution you want. You then work in that distribution you created, which is approximately isomorphic to the data. But other models do similar things, just more lossy but better at things like sample generation. That's the tradeoff, interpretability/density vs expressitivity/sample quality. But diffusion and NODEs/Score models are closing that gap. But you're going to need to look at applied papers to view more people using them in ways like operational vector spaces. For example, there's VITs TTS uses a NF to parameterize parts of the model or controller networks tend to use similar things. It's more about thinking how your network works and communicates. I think a lot of people are just not thinking to hack away and operate on networks as if they're mathematical models instead of a locked box.

Reply on news.ycombinator.com