Ask HN: Non-Malicious Bots

When I ask an AI to load a link, sometimes it can load it, and sometimes it can't (and I have to copy paste the text myself).

Sometimes it can load it even when it has the "checking if you're a malicious bot". I'm wondering how that works. Isn't AI scraper traffic literally the thing that page is designed to stop?

Do they let Google, OpenAI etc. user agents through? Or do those companies just have a bunch of guys in a warehouse somewhere with Chrome open, forwarding the AI's HTTP requests?