Hacker News Reader

If I'm building a search engine, say focused on telescopes, can I use 80legs to crawl youtube videos (or results from google video search)? I mean can I add this url to my list of urls to crawl? - http://www.youtube.com/results?search_query=telescope

View on HN

Our default crawler obeys robots.txt and it looks like the /results URLs are not allowed. However.. I think you could start from a URL like http://www.youtube.com/watch?v=sAzhOSbxMiI and then crawl to the linked videos from there...

Oh I didn't know about the robots.txt rule for /results. Thanks for pointing it out or I would have built my own crawler and got banned. I think I'll go with playlists.

Reply on news.ycombinator.com