Hacker News Reader

But whats the point of it downloading the robots.txt file if it isn't going to honor it?

Presumably, one cluster (or whatever google calls them) is honoring it, while a different cluster isn't.