Hacker News Reader

This is so far beyond me, so I’m going to ask and hope someone knowledgeable is here:

Could this idea of reasoning through fundamentally different (it sounds like?) models be used to link even more distant models or systems?

If you can make a small leap, then you can construct bigger leaps through a chain. What else can latent space link to?

Instead of tool use through text, tool use through analog signals?

Instead of language prompts, analog sensors?

View on HN

That sort of thing is common. Image generation models link a small LLM-like model like CLIP with an image oriented neural network, that's why they can go directly from words -> images with no tools in between.

In principle it can be pushed a long way. I've noodled around with the idea of retraining an LLM to directly output compiler IR, allowing it to code at (in principle) many times the speed of existing coding LLMs. But understanding the results gets harder, not just in the mechinterp "what do these activations mean" sense but in the sense that it'd be emitting a form of code that doesn't use words at all.

They’re not fundamentally different at all. In fact the paper is using existing models as a backbone and just providing a shortcut between the two ends of the pipeline

Reply on news.ycombinator.com