Good point! Thanks for the feedback.
Let me add that to my todos.
Let me add that to my todos.
> It has an extensive proxy IP and retry system in place to bypass bot detection.
Seems like a bit of "talking out of both sides of your mouth".
> Good point! Thanks for the feedback.
> Nothing like this will be added to the product. Money comes from scraping content and thus content will be scrapped regardless any non-scrapping hints and we will be actively working on countering anti-scraping measures.
I don't say that makes them bad, they just knew other things, so I can totally not have my mind boggled that someone launched a product like this and didn't take obeying robots.txt into consideration and then adds it to the todos when someone complains.