If you could track hand movement 1:1 enough that interacting with a 3D space was as natural as it would be as if it was actually in front of you, the problem is definitely a solved one at that point. If Apple manages to do this at any point in the lifecycle of this product line, it'll be a significant breakthrough for this type of computing / experience
Further, you can absolutely make 3D objects able to be grabbed, thrown, resized, etc. and it is actually pretty simple to do.
This is a button which has no physical presence at all.
Maybe Apple deliberately decided against including optional controllers, to make it clear that they are aiming at AR, not VR gaming. But even in AR the lack of tactile feedback could be jarring.
This all seems like playing Quake on a touchscreen.