But if you label it, why wouldn't you start labeling all other sorts of ridiculous things?

GMO vs not-GMO is irrelevant information for the consumer outside of marketing and ignorance-based paranoia.