TidyFinder is the first step towards that goal. I'd appreciate any feedback/criticism/suggestions that you can offer. I value the HN community's thoughts. Thanks.
TidyFinder is the first step towards that goal. I'd appreciate any feedback/criticism/suggestions that you can offer. I value the HN community's thoughts. Thanks.
App and scrapers are written in Python with MongoDB storage. I was pleasantly surprised with MongoDB's query speed given I only index keywords. There are a small number of products in the database, but its still very fast.
To get the data I've created scrapers for each site. Currently, only Target and Amazon are implemented (with Container Store soon to come). Target and Container Store provide a product feed when you sign up as an affiliate. They FTP their product info to your site and I use that to parse the links. Using each link, I then scrape the page to store the relevant information for various products in the database. Amazon provides a nice API and there is a Python wrapper which makes it easy to query (python-amazon-product-api).