You can find just a few sources below, out of the MANY studies done.
The gist is: fat is not bad; at all. Most of Vitamins contained in Milk are fat-soluble. That means that all the good properties of milk are transported by fat. Also, whole milk is less processed.
Here's a very "intuitive" reasoning, as an exercise: if Vitamin D is fat soluble... how is it possible that Skim milk has MORE Vitamin D (or A) than whole milk? (answer: because it's artificially added).
Sources: