I find it quite concerning that we are going into a situation where some things can crawl some sites but not others.

I have my own serious ethical concerns about AI, but from a pure searchability standpoint it is likely to cause more problems than solve.

Information is going to start being silo'd. It would be like if I had to go to Google for one thing and Bing for something else.

That being said, I am also quite concerned about what you mentioned. I was not happy when I found out that ArsTechnica signed a deal with OpenAI (or I assume their parent company did). Did that deal include any requirements of how they talk about OpenAI, AI in General, or their competitors?

The concerns over AI are real, but they are putting a short term gain over the long term damage by somehow thinking that licensing data is going to save them.

The tone of this is a bit weird, do we have similar "blocking google", "blocking anthropic", "blocking meta" articles? Instead of what is really happening is those partnerships were formed so everything not that partnership is being blocked.