I am slightly intolerant and have symptoms if I stuff myself with raw milk. Other than that it is no problem. I have a friend with the genetic condition and even he drinks milk. He does get problem quickly, but not if he drinks coffee with some milk or yogurt.
I think all people develop an intolerance with age naturally, but people that consume much milk will have that process slowed down. I think the genetic factors can just accelerate the process that is responsible to induce intolerance.