With bash's HISTIGNORE, I can consciously prefix my command with a space to prevent it being added to history.

ls I usually don't care about, but there are directories I regularly cd to, so it would be nice to have those in history.

I can think of a neat heuristic, which is that I often cd to an absolute or home directory, so if the path starts with / or ~ I'll possibly want to cd there again in the future. Changing to a relative path on the other hand, I tend to do more rarely and while doing more ephemeral work.