archive.org will only archive publicly visible content and it respects robots.txt
Not since 2017. https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
They now have a clunky manual process to exclude your site. https://help.archive.org/hc/en-us/articles/360004651732-Usin... ("How can I exclude...")
They don't spoof user agents, but blocking them actively doesn't remove their history.