Have you tried Zathura (http://zathura.pwmt.org/) ? It's a pretty lightweight PDF viewer with vim-like controls.