We already have a standardized system - robots.txt - and AI bots are already ignoring it. Why would more standardized headers matter? Bots will ignore them just as they do today, pretend to be regular users, and get content without paying.

(A secondary thing is that AI bots have basically zero benefit for most websites, so unless you are some low-cost crappy content farm, it'll be in your interest to raise the prices to the max so the bots are simply locked out. Which will bring us back to point 1, bots ignoring the headers)