Not having controllers will be pretty limiting for gaming - some types of games will be fine, others definitely won’t be.
For everything else games could use hand tracking with a similar control scheme to Kinect and Wii attempted.
However, most VR games that require a controller for VR interactivity with "real world objects" tend to fare less well as they create a cognitive dissonance between the actions the player is taking and their perceived results. Anecdotally, for the few VR demos I've tried I agree.
Yeah, it will be limiting for the preexisting games built for preexisting VR systems. It will also be perfect for the types of games people envision and build for this generation of device. Not recognizing that seems more like a lack of imagination on your part than a failure of design on Apple’s.
Still, sometimes you need buttons. While mobile gaming is obviously huge there are certain types of games that just don't work all that well on it. In the gaming space Apple will be directly competing against other VR headsets designed for gaming. Mobile games don't really directly compete with consoles or PCs.