Well, if it's a CLI based server I'd doubt X would be needed at all.
OpenBSD Amsterdam (A well known provider in the circle of OpenBSD users) doesn't enable X by default on its autoinstall file:
https://openbsd.amsterdam/setup.html
EDIT: It enables xbase.tgz but not the rest.