Anyone Experienced with Crawling Websites? | Hacker News Reader

Anyone Experienced with Crawling Websites?

7 pointsFlash77555 comments

Hello fellow hackers,

I am trying to create a tool for myself that can crawl a few websites that I usually go on to compare the price of the same item. They have no APIs.

A couple questions:

1. is this LEGAL? 2. if I am crawling, what is the best way to approach this? does each website's crawling mechanism have to be manually written since they are unique or is there some strategy for scale if i need to expand the number of sites I crawl through in the future?

Thank you!

-F75

View on HN

I’ve done a good amount of it and it is always a pain. I’ve never setup broad scrapers that index tons of sites, but instead always a select few that I wrote specific crawlers for.

1. The act of scraping public pages was declared legal (at least in the US). What you choose to do with that data may or may not be legal. Comparing prices shouldn’t be an issue though IANAL.

2. As someone else suggested I’d start with scrapy. It can easily take a sitemap.xml and start chugging along. You’ll need to build the specific scraper for each site. The browser dev tools will become your best friend. One thing to look into is the network calls. Rather than loading static content, they may be loading JSON from their servers and those are significantly easier to pull data from.

1. The precedent (so far) is scraping is legal if the scraped data is publicly available[A].

2. I guess the best approach depends on what data you're scraping. Some data it's fine to first convert to plain text, then scrape scrape that.

For structured data like tables and HTML, you're better off using the structure of the HTML itself.

I suppose you could design a framework that covers all the common tasks, then feed the framework parameters for each site.

It's not just handling different sites: the same site will change over time, and there will be oddities between pages/items on the same site.

[A]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I'm running a dashboard on an old samsung phone for crawling some sites and displaying them in an easy to read format.

Copy any request you made on chrome dev tools as curl convert curl to a function node. If anything breaks on site side just make a new curl and copy thoose values.

You can use an open-source tool like this one: https://github.com/dgtlmoon/changedetection.io

I would recommend using scrapy.

Reply on news.ycombinator.com