You actually have to consider what's the body going to do with the excess Vitamin D that can't be absorbed (because of the lack of fat). The answer is: probably going to stress more your kidneys.
For me, the bottom line is, why would I drink skim milk if it's just a more "watery" version of the perfectly healthy whole milk?
Btw, and this is just me, I'm not lactose intolerant (I eat A LOT of yogurt, milk, cream, cheese, etc), but skim milk makes me "sick" (lactose-intolerant symptoms). Maybe it's just my brain.