Imagine your startup is unwelcome at VC events because you're considered a coward after declining a duel. Or if you're an employee, you're passed up for promotions on the same basis.
You can think it bogus all you want, but that is small comfort if there are concrete socially-enforced ramifications for declining a duel.