I am now using https://github.com/ajeetdsouza/zoxide (in Rust) and it works really well.