I think the most likely problem is that by trying to be clever with your web server configuration you'll accidentally introduce an insecurity at your end.

On my personal webserver, the vast majority of "rando" accesses are from search engine spiders that I've never heard of (as well as the ones you have heard of). These spiders really ought to be secure against anything a server can throw at them, and if they aren't, that's their problem.

This whole comment section has me wondering if an internet-connected tea pot should respond to all requests with a 418 status code.