How about a joke in the context of saving money that said "Must've been done by a Jew" (my supremely limited interaction - with one person from Israel - was about him trying to get more money from the organization I was representing), would it be bigoted to point that out?
For my taste that "must've been a woman" line was cringeworthy. He could've said "Well I guess someone doesn't want to grow old".