Honestly, it's all about context and really the people say "oh he used female workers instead of women workers" what a pig are the people who are going to be mad no matter what they're literally just looking for an excuse.
And quite simply I find that demonisation of the word female to be rude and fundamentally sexist. We wouldn't have a problem saying "two males walking around" but in escence you would have a problem with "two females walking around". The fact being female is considered dehumanisng is a problem in itself and sexist. Being male is not dehumanising. See how that got turned around? You had good intentions then I got mad and used your words againist you.