Others have commented there is a perception that male cleaners will steal valuables. And it may very well be statistically true, given men's much greater criminal behaviour in general compared to women.
Women and men are equally likely to shoplift, so I don't think there is a difference there as shoplifting is the closest you'd get to sneaking out a valuable from someone's home.
You could make this exact same argument to justify racism in hiring cleaners as well, so probably not one you want to make.