That’s your problem right there. You can’t (or, rather, don’t) “VNC” to Unix servers. For normal remote access, you use SSH. If you really need to run a graphical X program continuously on a server, you run a virtual X server program like “Xvfb”, and point your graphical X program to that. You can then “VNC” all you like to this virtual X server. All without a monitor or even a graphics card or a mouse on the actual server.
This thinking – the idea that a server has something one can refer to as “the” screen, or “the” mouse – is completely and utterly wrong, and betrays a thinking born from Microsoft Windows and other desktop-only-oriented operating systems.