Hacker News Reader

> Instead of blocking, it would be neater to present and alternative version to the crawlers

IIUC, if a site presents different (view of) content to the crawler than users, the site can get de-indexed.

If that's a concern, and you're only worried about AI crawlers, then that's not a problem. Only provide the bogus pages to the AI crawlers, not to the search engine crawlers.

Assuming there's a difference, anyway. I suspect that with Google and Bing, there isn't.

Reply on news.ycombinator.com