Hacker News Reader

Have you ever heard of https://commoncrawl.org/? Maybe it helps you to bootstrap.

It's cool to see some new things happening in the search engine world.

I've considered that but didn't really know anything about the organization and how long they'll be around. We're gonna give Colly a try: https://github.com/gocolly/colly.

Reply on news.ycombinator.com