I've worked in these sorts of communal environments for the past six years and never had a problem. I happen to have friends and family who I have occasional disagreements with too. It's just eating together. Can't make it one day? It's not a mandatory meeting or something.
Maybe you just don't work well with that?