Yeah wtf? Everyone let's Google crawl their site but it's naughty for anyone else to do it... such bullshit!
"it's naughty for anyone else to do it" if that everyone are companies that will be using or monetizing that content without providing any kind of attribution or compensation to the source/creator.