Earlier in the chapter he says:
> On the contrary, I believe strongly that the color of a man's skin or the religion of his parents is, by itself, no reason to treat him differently; that a man should be judged by what he is and what he does and not by these external characteristics. I deplore what seem to me the prejudice and narrowness of outlook of those whose tastes differ from mine in this respect and I think the less of them for it
The chapter is discussing the appropriateness of government regulating negative versus positive harms and he makes the case that regulating negative harms is a short-sighted and ineffective means of achieving the policy aim of protecting minorities and that "fair employment" as imposed by the majority is not "free employment". In the same chapter he argues that "right to work" laws interfere with the freedom to engage in contracts. The arguments are far more subtle than Edward Zitron is making them out to be.
Importantly Friedman here and elsewhere refers to people who discriminate on the basis of skin as ignorant racists.
Throughout the book he makes the argument that if you remove the legal frameworks that enforced segregation, that people would over time choose to integrate as they found mutual advantage in doing so. Reasonable people can disagree with this assessment.
He finishes the chapter with this quote:
> On the other side of the picture, we should not be so naive as to suppose that deep-seated values and beliefs can be uprooted in short measure by law.
If anyone thinks that Friedman was a racist, they haven't honestly engaged with his writing.