For this particular case, I use a ncurses based file manager: ranger
Since I also love ranger as a file manager, I wrote my own integration of both: https://github.com/fdw/ranger-autojump . It remembers where I went in ranger, and I can also call autojump in ranger.