And yet despite centuries of use of singular they/them pronouns, people respond the same way when they encounter them. Presumably you are comfortable with singular they, given its long and frequent use?
If another cis man was suddenly uncomfortable with "they", I'd attribute that to some fragile sense of masculinity or a weird anti-woke crusade, as was the case with most such cases in my life, and find it personally annoying.
I don't want to deliberately make other people uncomfortable with my language, but it does feel exhausting to second-guess yourself, having to double-check if any innocuous phrase might be misinterpreted as an underhanded aggression, preempting a correction we know will be awkward for both sides.
I think we'd do better to consider context more than the specific language...