Well I mean it's my IP so I can open source the particular file if it makes you feel more comfortable. It's not closed source because it's secret, but because it's harmful in the wrong hands.

Web crawlers are harmful to independent web crawling. It can be very annoying for webmasters when misconfigured, and abuse leads to honest crawlers encountering obstacles. It would probably spell the end of independent web crawling if every tom, dick and harry with a few spare raspberry pis set up their own search engine operation.