Hacker News Reader

Comment by latchkey | Hacker News Reader

Start with a robots.txt?

https://www.codeium.com/robots.txt

Update: I also see that you serve your site off GitHub... so that might be part of it as well? Bing might not index those sites?

View on HN

No robots.txt is basically the same as the default contents of a robots.txt saying "All bots welcome"

No robots.txt is unpredictable because it depends on the implementation of the bot.

A missing robots.txt has always meant "allow all", so it would be really weird for this to be the cause. Robots.txt is primarily for disallowing (or really, discouraging) crawlers from crawling certain paths.

Why would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?

You're certainly getting a lot of free SEO help today. I'm not the one who has problems getting my site indexed, but it seems like they are different things.

  # Allowing all web crawlers access to all content
  User-agent: * 
  Disallow:

https://moz.com/learn/seo/robotstxt

At least you'll be able to look in your logs and see if BingBot is grabbing the file or not.

Because this is the lowest-hanging fruit imaginable, why not take the 5 minutes to add a robots.txt and then use https://www.bing.com/webmasters/robotstxttester to confirm that Bing can fetch/parse it?

A day later, I see you added the robots.txt and it shows up first now.

Reply on news.ycombinator.com