Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
Or just sending it a tiny zip bomb, or even just some sh*t right out of /dev/random
Bots are pretty good at detecting and preventing this kind of issue. Dumb clients would still be affected.
/dev/random is much too expensive for this.
flat files, a tarpit, maybe zip bombs as the sibling suggested. Nothing calculated. I'd go with zipped Brothers Grimm