That’s a good point, I hadn’t thought of these cases.

On a second thought, I guess a lot of marketing content would also love to be crawled by anything that crawls…