Hacker News Reader

I imagine the ONLY reason is that Google don't have such an API that is exposed to the public.

There is an API via which you can describe your URLs (it's called a sitemap) and you can ping to Google your sitemap when its content changes. You can have multiple sitemaps and ping only your changes. More on www.sitemaps.org .

But Google reserves the right to crawl non-sitemaps URLs, for obvious reasons. It would be quite a bad decision for them to restrict their crawls only to API-provided URLs.

We try to give search engines hints with the update frequency in our sitemaps.

We re-build our sitemaps nightly and make sure that new or recently-updated content is listed with an update frequency of "daily" or "weekly" and all other content pages are listed as being updated "monthly."

To be honest, I've never measured if it works, but it can't hurt.

How do you ping only changes? Don't you have to resubmit the whole sitemap?

You can have multiple sitemaps. You can ping just 1 sitemap containing only the links that you want to notify. You can use the optional <lastmod> tag to indicate a URL's last change date.

There are services in Webmaster Tools for pushing sitemaps, no? Which would not be quite sufficient for this--Google still has to scrape the respective pages in the sitemap--but it's about as close as you are going to get.

Reply on news.ycombinator.com