I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” However intelligent an internet trained model may be, I strongly believe you have to have some interaction with the real world to learn more complicated actions. So far, Pick and place, is all that’s been shown with the bigger models, so my hypothesis seems to be holding for now.
https://general-pattern-machines.github.io/
https://wayve.ai/thinking/lingo-natural-language-autonomous-...
It's just the straightforward application.
>I have a quote I like in this context, “Could you learn gymnastics just from watching videos?”
It's not like language models learn by some sort of magic osmosis. They're not just "reading" text or "watching" images. They learn by predicting, failing and adjusting neurons based on the data. Text is their world and they are interacting with it.