> Avoid detection with built-in anti-bot patches and proxy configuration for reliable web scraping.
And it doesn't care about robots.txt.
And it doesn't care about robots.txt.
Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.
Yes. It is. You've just made an arbitrary choice not to define it as such.
You're creating the wrong kind of value. I really hope your company fails, as its success implies a failure of the web in general.
I wish you the best success outside of your current endeavour.
I will add a PR to enforce robots.txt before the actual scraping.
[0]: https://github.com/lightfeed/extractor/blob/d11060269e65459e...