Didn't know that. That's handy, thanks!
Firefox also has the Vimperator plugin that gives vim-like nav for Firefox. Highly recommended if you like Vim.
I'm aware of Vimperator, but have never tried it, as I am an Emacs guy and about as useless with vi as I am with wild animals. It might be worth a look, though, if it does everything I mentioned above. (Does it?)
It does have a bypass mode for javascript heavy sites like gmail, with builtin shortcuts. Some sites present issues with link detection, but vimperator is highly highly customizable, so given enough effort you can make almost anything work.
There is also a chrome plugin called vimium that has a smaller feature set.
I also found this: http://conkeror.org/ But I have not tried it. It appears to be a customized version of firefox with emacs style shortcuts.
There are commands built in for a lot of actions, and if there is a right click option that the extension doesn't provide, you can always write a script to extend it. Vimperator is nicely extensible.
I'm not sure if Vim scrolling with hjkl resolves the issue you saw with the arrow keys- I'm sure that's site dependent, and you'll have to try it yourself.
Good luck!