Hacker News Reader

Seen a number of similar comments recently, though none tend to mention looking at raw server log files and checking for bingbot et al actual requests to pages. Would be good to confirm the actual sighting of the bot especially due to the likes of Cloudflare operating in the middle of the request.

View on HN

Through Bing Webmaster Tools, I can see pages being crawled, but not indexed. Bing has the sitemap and all the URLs. The robots.txt allows all crawlers. Even if there was an issue there, it would be on their end, and they should be the one fixing it. The result is the same: a mysterious disappearance from their SERPs. And no way or clue on how to fix it.

No doubt they'll attempt to crawl and their webmaster console will say so, I just think it's important to verify the request actually made it to your server via your own server logs in cases where you're using a WAF like Cloudflare, just to ensure that Bing saw what you intended to serve them. If they're reporting 4xx errors, it could just as easily be a WAF giving them the code, or your server but judging by your response, it wouldn't be your server. The server logs would explain away some of the issue.

Indeed. Thanks for the help!

Reply on news.ycombinator.com