Why does Google think it's a good idea to make that the case even if you don't block their crawlers?
Reply on news.ycombinator.com