I drink (black) tea with milk. So here's a _very_ anecdotal/hypothetical thought: my brother switched to drinking black tea without milk. A couple of years later, he had a kidney stone; and the doctor claimed it was the tea.

Some of the tannins that give the tea its color are small enough to get into the blood stream, but get caught in the kidney, possibly leading to stones. So if you add milk, the big fat molecules of milk trap these, which prevents them from getting caught in the kidney; hence no stones.

I could be wrong...