In modern vim you do have direct access to the arrow keys. Rarely is this meaningful, for someone skilled in vim.
It can potentially be faster if you need to move just a couple characters over, but if you need to move 25? It winds up being something like: Esc 22hhhh
People navigate by jumping across words, across features, searching...