Driver support isn't a given. There's lots of normal hardware which may never be server qualified. You can make a desktop that will run server fine, but it's kind of a Linux situation; you can't pick random parts and assume they'll work.
Another issue I've run into was Server 2016 not having trackpad drivers for a Dell laptop's I2C bus/trackpad.
I suspect power management is also poor.
On which version? I'm running Server 2016 (Datacenter) with an Intel iGPU, and I don't recall having to do anything special to get it working.