If I read this right, surely the point is that he was not crawling the /Wiley/ site, but rather something else that contained those links? This is the thing about deliberate contamination - it spreads.
He states that he accessed the site from the library so that he had access to Wiley online (as part of the Library's access arrangement). And he states that he was building tools for automatically linking documents. (tool, not browser).