Once upon a time I had to manually disable ROBOTS.txt parsing in one crawler just to stress test a staging machine for a friend.
The lesson from this is that ROBOTS.txt works as long as everyone follows a line set in sand.
The lesson from this is that ROBOTS.txt works as long as everyone follows a line set in sand.
No comments yet.