Sure that works if you’re selling perfume or handbags, but if you’re a family restaurant then rich people and regular people eat the same quantity of food, your margins are low, why accept it?
Moreover Amex is pretty rare outside of certain markets. If only foreigners use Amex, and your business is not catered to foreigners, then you probably aren’t going to support it or in some cases even know what the hell it is.