like, take existing bookmarks, make a crawler.
nice work, btw
The hardest part BY FAR is the crawler: initially I was using Apache Nutch but it got slower and slower as the index grew, so I replaced it with my own crawler that I wrote in PHP (comfortable for me) and made that multi-threaded using Supervisor.
The second hardest part was the amount of security I had to build in to prevent bots running spam searches and hogging my infra.
I'll try to write a blog soon and post it here.
1. Always identify your crawler via a consistent user-agent string, that explains its a web search crawler and not a generic web browser.
2. Always obey the directives in robots.txt.
3. Make sure your crawler is not too aggressive (low frequency of requests).
(updated for formatting)