The robots.txt exlusion loophole has been known for quite a long time.
Yes, but it seemed like they had changed their mind, exactly because there is a huge issue with "expired" domains, see:
https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websi...
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
They experimentally ignored robots.txt on .mil and .gov domains, and I thought they were going to extend this new policy for all archived sites.
The situation/status is not clear, though the retroactive validity of robots.txt remains (at least to me) absurd.
It is IMHO only fair to respect a robots.txt since the date it has been put online, it is the retroactivity that is perplexing, as a matter of fact I see it as violating the decisions of the Author, that - at the time some contents was made available - by not posting a robots.txt expressed the intention to have the contents archived and accessible, while there is no guarantee whatsoever that the robots.txt posted years later is still an expression of the same Author.
Most probably a middle way would be - if possible technically - that the robots.txt is respected only for the period in which the site has the same owner/registrar, but for the large amount of sites with anonymous or "by proxy" ownership that could not possibly work.
This would also solve situations where a new owner blocks robot access for a domain where the former owner is OK with the existence of the archived site.
It seems to make the most sense to only have a robots.txt affect pages archived when that specific version of robots.txt is in effect.
At the time of crawling the robots.txt is parsed anyway.
If it excludes part or the whole site from crawling, it should IMHO be respected, and the crawl of that day should be stopped (and pages NOT even archived) if it doesn't then crawling and archiving them is "fair".
The point here is that by adding a "new" robots.txt the "previously archived" pages (that remain archived) are not anymore displayed by the Wayback Machine.
It is only a political/legal (and unilateral) decision by the good people at the archive.org, it could be changed any time, at their discretion, without the need of any "new" syntax for robots.txt.
But that is easily achieved by politely asking the good people at archive.org, they won't normally decline a "reasonable" request to suppress this or that page access.
As a side note, there is something that (when it comes to the internet) really escapes me, in the "real" world, before everything was digital, you had lawful means to get a retraction in case - say - of libel but you weren't allowed to retroactively change the history, destroying all written memories and attempts like burning books on public squares weren't much appreciated, I don't really see how going digital should be so much different.
I guess that the meaning of "publish" in the sense of "making public by printing it" has been altered by the common presence of the "undo" button.
Another sign I am getting old (and grumpy), I know.
In pretty much all other cases--except where they were public domain or CC0--it's probably not strictly legal to archive them at all. Therefore, it makes sense to bend over backwards to remove any material if asked to programatically or otherwise.
>I see it as violating the decisions of the Author
Maybe in some cases. But, for better or worse, preventing crawling is opt-in rather than opt-out, and defaults are very powerful. You didn't explicitly tell me that you didn't want me to repurpose your copyrighted material isn't a very strong legal argument.