Google is quite scrupulous about robots.txt.
There's plenty of hypocrisy at that company but this doesn't strike me as a case of it.
There's plenty of hypocrisy at that company but this doesn't strike me as a case of it.
I'd suggest respecting Google's robots.txt as much as Google respects our DNT settings.