robots.txt is not legally binding, neither are the "terms" on a website when it come to screen scraping or automated access.

A "robots.txt for AI" would be nothing other than a polite request that will be ignored by the vast majority of organisations.

Under the current understanding of the law and copyright, the only preventative measure is putting content behind a wall with an explicit user agreement to access it. Effectively, if it's readable by a human without having to actively agree to a license, it can be scraped and used for any purpose, as long as it's not reproduced verbatim.

What we need is a better understanding of Copywright and data mining in law. We need test cases.