Another thing we noticed is that google's crawler is kind of aggressive, I guess they are in a position to do it.
Our paper in case someone is interested: Optimizing Apache Nutch for domain specific crawling at large scale (http://ieeexplore.ieee.org/document/7363976/?arnumber=736397...)