Mupdf on linux is both lightweight and vim-like
I used mupdf for years. Then I discovered llpp (a wrapper on top of mupdf). I never went back since, this is my favorite pdf reader by far :)
also, clicking on a listing gives details about what is vim-like for that item