I have been writing bots and scrapers for almost 20 years.
Not once have I had someone I work for in this capacity even ask about robots.txt
Not once have I had someone I work for in this capacity even ask about robots.txt
Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will.
So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism.
In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else.
Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with.
I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
But, because this person could be anyone, anywhere your words are hollow and empty. You are impotent to stop their future actions.
The internet is not a society. The internet is all societies. Hell, more and more it's not even human societies.
And may be or could be part of the solution by them taking upon themself to implement it in an ethical manner by having the bots and scrapers respect robots.txt; unless the spec specifically says not to in that case I agree with finding new clients or another job
The internet is built on scrapers though - Google is a scraper, and an opt-out one at that.