Thanks! I don't why they don't limit Google, Bing crawlers to the homepage via robots.txt. Then crawling should not overload an already busy website. Honestly, it seems like an overreaction to me.
For what it's worth, pg said that "crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory" (at https://news.ycombinator.com/item?id=3277661).