Hacker News Reader

Comment by Freak_NL | Hacker News Reader

It looks good and seems to work with some arbitrarily picked recipes on the usual large recipe websites, although more obscure links cause some bugs¹. Is the source code on GitHub? Are you handling specific websites such as allrecipes.com with bespoke code?

Technically the project is interesting, but if you want to offer a commercial API you might run into copyright and fair use issues (as with any scraping tool). Not so much a problem for personal use, but expect angry letters from the major recipe websites for violating their terms of use (i.e., this is a threat to their business model).

1: Try this link: https://www.thespruce.com/peking-duck-recipe-694920 . You'll see a lot of garbage HTML and XML entities in the JSON that can be filtered and replaced fairly easily.

View on HN

I'm actually amazed it manages to scrap minor websites. Scrappers usually have to take in consideration every single website they're targeted at, and they break each time there's a major redesign (or sometimes, a minor one).

@brad0 : how did you manage that? Was thespruce.com in your targets?

I don't know about this scraper, but one thing that mine does (http://www.copymethat.com) is to "read" the complete page looking for certain word combinations that indicate ingredients or steps. It also considers styles and location on the page and looks for keywords that tend to start or end a recipe. It then picks what it considers to be the strongest recipe on the page. This means that it can pick up some weird things if the page doesn't actually contain a recipe; It really wants to find one!

Interesting approach, thanks for mentioning it. I guess it means you have a lot of unsuccessful results? Do you try iterate several times on the same page to find different possible sources for a given info and rank them, or is it something more like "if we're not confident enough, forget about that info"?

There are hardly any unsuccessful results. (Assuming that the page actually contains a recipe.) People have copied recipes from more than 70,000 websites into their recipe boxes. Of course, I can't check that all the millions of recipes have been accurately copied, but we do check a lot and also get terrific feedback. The parser first goes through all lines/sentences on the page and gives them a rank based on whether it seems to be an ingredient or step. Then it looks at groupings (several steps together) and then the placement of the ingredients compared to the steps.

I'm also curious what the approach was to being able to scrape sites in a way where the code is not coupled to a known, fixed markup structure using css selectors, xpath etc.

What are your plans for the project? Will you be open sourcing the code on GitHub?

Reply on news.ycombinator.com