This is an awesome initiative. Is anybody else thinking of the Watchdogs Blume though? ^^ https://watchdogs.fandom.com/wiki/Blume_Corporation
176 karma · joined December 26, 2015
Mainly in use in the crawler: * request * cheerio
It's basically scraping all links on every page it hits and tests the headers if they are containing the clacks value.
I added the form so people can submit pages to speed up the development of the list, even though I believe eventually the crawler would get to their pages :D
I completely agree, I bet it's not a great job most of the time and a really different and isolated world.