Engineering household robots to have a little common sense
news.mit.edu
news.mit.edu
Even the humble Roomba has already been looked at as a spy for selling maps of people's houses (https://gizmodo.com/roombas-next-big-step-is-selling-maps-of...) and took photos of a woman on the toilet which were leaked online (https://www.businessinsider.com/roomba-photos-recorded-bathr...)
No way in hell would I bring a robot butler into my home unless I can be certain that it won't be collecting and leaking sensitive information about my life/household to others. We have too much of that going on as it is.
I would love to have a robot that can carry out this common household task. I have to do this every week, on Saturdays, when the bowl holding the marbles overflows with the offerings deposited by the Roc birds at the top of the windcatcher in my desert redoubt. The marbles are sometimes made of quartz, sometimes nephrite, rarely ruby, emerald, sapphire or diamond and so they have to be disposed off, or the sand worms will come for them.
More seriously, training a classifier to map LLM instructions to robot arm actions, as the researchers in the article did is smart, but their motivation was error correction and that, while intersting, is not the big problem. The big er limitation of imitation learning is that if you teach a robot to scoop marbles from bowl A and pour them into bowl B by imitation learning ... that's the only thing it will be able to do. Scoop marbles from bowl A to bowl B. The same bowls A and B, placed in the same positions, with only a small error tolerance (centimeters) and the marbles better be the same colour as the ones seen in training, the spoon better be the same shape, or else!
Or else you're going to be stepping on lots of marbles. My common sense tells me that can be very painful.
LLMS in their current state are certainly not the answer, but point to some exciting possibilities that we couldn't do before because of sheer complexity.
Less sarcastically, automated planners need domain knowledge to break up a task into sub-tasks, and the neural net folks don't like that because they are taught at ML 101 that everything shall be a) end-to-end differentiable and b) have no human in the loop. LLMs seem to encode some kind of domain knowledge, the "common sense" that the article's title mentions, and so there's various teams trying to get robots to follow instructions with LLM-in-the-loop. A year ago I think it was LLMs generating PDDL, now it's natural language labels that obviously need a translation.
This won't work though a) because it's trained by imitation learning and nobody cares about a robot that can scoop marbles from bowl to bowl but would need re-training to do the same thing with e.g. strawberries and b) the LLM itself isn't grounded anywhere and has no way to self-correct its self-correction, once it falls into error. This, too, shall pass.
Btw, are those people mad? Marbles? Couldn't they find something less painful? What happens when your robot overturns the bowl and there's marbles all over the floor of your lab?
Jeez I am bitter I bought that one. It is up there with my HTC Vive of worst expensive tech gadgets.
Reading earlier mathematical texts (eg, Euclid's Elements) and their "natural language" proofs reminds me that a picture is truly worth a thousand words!