> Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains.
The last clause is important and suggests there might be further advances from such joint training, incl from interactions in the real world.
The low-level policies are from
RT-1 (Brohan et al., 2022), a transformer model that takes
RGB image and natural language instruction, and outputs end-effector control commands.
For those that don't know, RT-1 (Robotic transformer) is previous work from the team that converts natural language to custom control code.
You can read more about Rt-1 here:
https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-...
Maybe I'm missing something, but this sounds quite generalizable.
This issue was a general disappointment in the robotics community; they had a LOT of funds to get robotics data with and they spent it on somewhat trivial tasks that we had almost solved already with much smaller and more principled models, instead of getting human demonstration data for more complicated tasks.
While I would love to see them working on robot chefs, I can appreciate that they want to start small. And regardless, it doesn't seem like there are any constraints outside of data for this architecture to work in more and more domains.
Except they are (were?) one of the biggest spenders in the entire field. While it may seem easy to hand-wave away "Just add more data!", robot data is way more expensive to get than language/image data, since people don't generate that naturally as they browse the internet. If their current operation was already too expensive for Google to keep running (as evidenced by Google shutting down this research arm), imagine what would happen if they proposed "let's spend a couple more orders of magnitude to get data!"
All I am saying is that they are going for an unambitious project that looks cool to the outsiders, with buzzwords like LLM. There are work by others [1] that are much more impressive in terms of manipulation diversity that would probably be a much better bet to add more data into, since they show a much more promising route to the future.
[1] https://twitter.com/chenwang_j/status/1628792565385564160?t=...
Their software efforts still seem to be going strong, and I don't really think it's fair to say that demonstrating transfer learning in an embodied model none the less, is unambitious - nor does that seem like it's reflected in the results - but to be honest, I'm more or less a layman when it comes to this so I'll defer to you and keep what you're saying in mind.
To that end, if you're still feeling up to it, maybe you could tell me what your thoughts are with efforts like this from Google:
https://diffusion-rosie.github.io/
Seems to compare (somewhat) to Mimicplay, in that it is attempting to create more data for "cheap", even if it's not "real" control data.
I think a critical piece is being figured out as more and more such research emerges.
IMO the logic making the decision of what to do is by definition controlling the robot, even if it’s intermediated by subsystems and electrical components that translate commands to movement.