I was the exact same until I started using relative line numbers. Then I can just look at a line and see that it's N lines down and jump there immediately. With j or k repeat I'll often over shoot and then have to go back which is kind of annoying.
Relative numbers are useful to save a few keystrokes. But in every other context, absolute numbers are what's relevant.