Dalai Lama was even cancelled by twitter users for that. He said that his successor, if a woman, must be beautiful (since she would be working with people and people care about looks). An appalled journalist asked: "but isn't the internal qualities all that matter". Dalai Lama mentioned the necessity of balance infuriating some of the viewers who then cancelled him.
Reason why your mentioned path is wrong it that along those lines one can arbitrary choose min and max limits of the function and choose the middle. So to give a very crud example "if murder is not ok, than acceptable number of murders is 0 and on the other end we have 8 billion people so middle is 4 billion?"
In same sense your version says that between slave and aristocrat there are many levels of gray, but in fact slaves are just a slaves and aristocrats are aristocrats, being in the game just helps aristocrats and prolongs the cycle of suffering.
Some people got angry at him? Did he lose his Twitter account? Did it cost him his job? Is he not the Dalai Lama anymore?