The post says that the imitation network takes the input from the vision network and processes it to infer the intent of the task. Isn't the "intent" always "to stack"? Or can the imitation also be just re-arranging blocks in another configuration?
This part is interesting, if I understood it well.. > "But how does the imitation network know how to generalize? The network learns this from the distribution of training examples. It is trained on dozens of different tasks with thousands of demonstrations for each task. Each training example is a pair of demonstrations that perform the same task. The network is given the entirety of the first demonstration and a single observation from the second demonstration. We then use supervised learning to predict what action the demonstrator took at that observation. In order to predict the action effectively, the robot must learn how to infer the relevant portion of the task from the first demonstration."
Does this mean that the imitation network has been trained on stacking, un-stacking, throwing...and other such tasks, and then it identifies that "stacking" is what is being done in order to imitate it?
Is there an ELI5 for what the 2 NNs are actually learning?