I find relative numbers distracting (because they change), and don't use them, but that's definitely not a problem in Vim: you can jump to a specific line very easily by typing e.g. "123G", regardless of whether line numbers are visible.
Excuse me, I meant when one needs to know in which line he is.
You should upgrade to patch 787 or above if you can: