A mouse's strength comes from its ability to encode movement. A keyboard can do this also, but it requires many more inputs to achieve the same result. Consider that for every unit we wish to move a pointer, we need to either press a key once, or move a mouse 1mm. If I wish to move the pointer 100 units, I need to press a key 100 times. Whether I choose to do this repeatedly or by using the system's key repeat feature to make it happen, it will take some time. With a mouse, I just need to move the mouse 10cm. One movement of the mouse is worth 100 keystrokes. Increasing the efficiency of the keyboard requires added complexity. In the case of VIM it might translate to something like "100j". But continually typing "100j45k10l" etc is not beneficial for fast paced FPSs.
A mouse's disadvantage while editing text, is the requirement for both hands to be at the keyboard at the same time in order to type anything moderately complex at a reasonable speed. For a user with a taste for speed and efficiency, the act of moving a hand to the mouse and back can take what feels like a lifetime. VIM's power isn't in its ability to point better than a mouse - it is in describing the actions required to modify text in a manner so efficiently that the act of using the mouse becomes less efficient.