There's a significant mental leap here. "I block these IP to conserve my resources, therefore they belong to clueless or malicious organisations". It's wrong in both directions:
* I don't think Google, Bing and other crawlers are inherently malicious, and certainly not clueless. Search engines serve a very important role in the internet. Ditto archive.org, and probably dozens of other bots.
* IP based blocklists work well for honest bots (not malicious, or at least not illegal). Malicious bot operators just buy SIM cards and use regular mobile internet for the crawling (basically unblockable, because the IP may be renewed every day or every hour). And the really malicious actors use residential proxies, i.e. botnets that proxy traffic through normal users' computers. Anyway I wonder how many of those 56MM IP addresses are regular dynamically allocated consumer grade ISPs.
>1-5-2024
For the love of all that is holy, what is this date format.