Comment by Lammy | Hacker News Reader

Top New Best Ask Show Jobs

Comment by Lammy | Hacker News Reader

Are there any good FOSS projects that would let me run my own crawler?

YaCy is the only one I'm aware of. See https://www.yacy.net/demonstration_tutorial_screenshot/

YaCy is a decentralized search engine

Try Apache Nutch.

Looks like Common Crawl switched to this from their homegroan crawler. That's high praise!

isn't that what Searx is? https://asciimoo.github.io/searx/

No, it is a metasearch engine but not a crawler.

Reply on news.ycombinator.com