Hacker News Reader

Comment by nopinsight | Hacker News Reader

Current LLMs clearly have much coarser models of the world than humans do. Their training data do not include as many modalities and training data with spatiotemporal dimensions are inadequate.

Up and coming multimodal models are changing this.

View on HN

"Multimodal" models only promise an additional layer of indirection and interpretation from the world through sensors into the same conceptual mechanism. An image is converted to tokens, a sensor is converted to coordinates, and it is all fed back into a correlating model. But this isn't adequate either. The famous dress illusion picture has RGB values closer to gold and white, but the actual physical dress was black and blue. The difference between perception and reality is something we don't have a handle on when it comes to these models.

Tokens are just another representation of the image, like 3 matrices or electrical impulses in our brain. They probably encode most of the relevant information, the question is whether the linear algebra at the core of a transformer is good enough to use this data as efficiently as we do. I don't think we can currently say if it is or isn't adequate.

> There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.

I was responding to the point in your first paragraph in your upper level comment and this quoted paragraph. Multimodal models will address that.

Your new point about perception could still be a problem for multimodal models, but it’s also possible that with sufficient data, the new models could address that (eg it could say people perceive the colors differently, as A and B).

Multimodal models map between human language and raw sensory data. It’s fundamentally similar, in principle, to part of how we learn our first language. (Not saying we do only that though.)

Reply on news.ycombinator.com