People say the same things all the time in companies where women complain about inappropriate sexuality. "What if I say something that offends her?"
Most of us are good, inoffensive people. We are saddled with growing up in a world that is changing. It used to be acceptable to have a topless woman on a calendar in your locker at the factory. Not any more. Men used to take each other to strip clubs for business entertaining. Not outside of Oracle.
We all deal with the fact that we're in the middle of change, and it isn't always comfortable. Just as it wasn't comfortable for people dealing with the acceptance of homosexuals: I think everyone accepts that the word "faggot" is no longer appropriate.
It would be far, far easier for everyone if they just, well, shut up. But these things are hard, and it's up to our generation to carve out new social norms so that our children will reflexively and automatically act in ways that don't offend anyone.
I feel for you, there's absolutely zero wrong with you having and expressing these concerns. My message to you is that we men can come together and support each other and be a generation that others look back on with respect, just as we look back on the generations that broke the colour barriers with respect.