While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up spending a disproportionate amount of time banning IPs from bots running poorly configured tools like this, which aren't rate limited and crush websites.

I'm grateful that "Obey robots.txt" is listed as part of it's standard behavior. If only scrapers cared enough to use it as well.