robots.txt was originally created to include such bots. That they think they don't need to respect it goes against the original intent.
Eg: https://www.robotstxt.org/faq/kinds.html >"What's New" monitoring
Eg: https://www.robotstxt.org/faq/kinds.html >"What's New" monitoring