Even if not based on popularity, I do think HN would be a great source to find URLs, from where we can continue crawling

Do you have any web crawling specialists I could speak with? Happy to write the code, just don't want to overload any servers