> From a computer vision perspective, I think most of that is fairly easy
No, not at all. The hard part is to do computer vision to understand what you can do with objects, not just convert object representation of images into string texts. For example, can I move this object away to get an object under it? Can I place an object on this object without it toppling over? Is this object dirty and moving it will smear things all over? If I topple this object, will it break? How much pressure can I apply to this object without breaking?
Those things are necessary to have an intelligent agent act in a room, every animal can do it, and our software models are nowhere near good enough to solve them.
You need the computer vision program to also have a physics component so it can identify strain, viscosity, stability etc on objects, and not just a string lookup table it needs to understand that naturally like humans as the same object category can have vastly different properties identified by looks.