what does the site get out of ai users? it doesn't seem like they have a strong incentive to let you not actually visit the site
shopping makes some sense if the bot can still buy things at least, but a lot of other ad dependent sites really have little reason