You don't need X to run OpenCL, only a driver. Pretty sure the same holds for OpenGL.