Hacker News Reader

I'd love to see what kind of scraping tech Facebook use. I've been getting increasingly frustrated with the python scrapers and ecosystem and have been checking out Colly. But I'm sure whatever they have built must be nuts.

I wonder if you could train scrapers with AI with a reasonable level of success.

View on HN

As someone who maintains a scraper [1] in the python ecosystem, can I ask what you find frustrating?

My scraper is targeted at smaller crawls, so your feedback might be totally irrelevant to me... but I always want to keep an ear out.

[1]: https://github.com/cldellow/datasette-scraper/

I haven't used your scraper, I worked with scrapy. I found debugging as well as command and control to be challenging. Colly was a big step up.

For reference I am primarily a Golang programmer so I find weakly typed tools frustrating.

Reply on news.ycombinator.com