Just say gentleman's agreement. It's okay to use a real phrase.
(I guess this is largely culture related thing, but e.g. among my peers (high educated 30s Europeans) gender-biased words actually cause more distraction.)
And anyway, I think you'd have been better off just saying, "I wish we could all agree...."