"Being part of the kernel" is defined as being in the kernel's source code tree. Not all of the kernel's source code is included in a kernel binary, most pieces are compiled as modules that can be loaded at run-time. However many distribution do indeed ship the binaries for most kernel modules so some server distros like Ubuntu server probably do have GPU drivers installed, although they are never loaded.