Ask HN: How to prevent duplicate content in a distributed webcrawler? | Hacker News Reader