If milk was that bad, humans wouldn't have developed adult lactose tolerance 10K years ago. Being able to drink/eat milk and dairy products was clearly an advantageous adaptation because it became a very widespread mutation.
https://www.smithsonianmag.com/arts-culture/lactose-toleranc...