I understand the point of using a special user-agent to crawl webpages for indexing, but search engines should use a "regular" browser UA string (full JavaScript, etc. to simulate an actual browser) occasionally. From a different IP range too, of course. If the contents of the page are wildly different, penalise the site.