Color comes from the initial neural network step. Since skin color is relatively predictable from facial features (ex: nose width), it should be able to do reasonably well.
Just imagine the kind of CCTV usage being discussed elsewhere in this thread. But the neural network happens to have a wrong bias towards skin colour...
One of the best articles I've read on the topic, if you're interested: https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59...