HNHacker News
TopNewBestAskShowJobs

throwaway12s34

2 karma · joined September 7, 2017

submissionscomments
throwaway12s34··on Show HN: Page.REST – An API to fetch details from a web page as JSON
How do you handle sites that have scraper prevention? Such as captcha, IP throttling, etc.

I'd pay $100-$500 per-month for a service that could reliably scrape some particularly difficult sites. With that being said, I'd need the service to be able to handle ~100 req/s in bursts and 2-4 req/s on average.