There are packages like image libraries, Java etc that rely on X11 libraries. The safe default is to have it around.

You can always choose to not install all X* packages. In fact, for a server oftentimes you don’t need more than the base package (you barely need a compiler either or games).