I guess sometimes Google does this kind of crap just to prove (mostly to themselves) that they can. Truly, the bully of the internet.
I guess sometimes Google does this kind of crap just to prove (mostly to themselves) that they can. Truly, the bully of the internet.
However, it's clear that for an archiving effort this big, people at Google are explicitly allowing it. The user agents and fetch patterns of the Archive Team crawler were clearly distinct enough to get caught by an automated tool, and someone knew someone at Google in order to get it unblocked.
Unfortunately, any archival effort that requires the "Warrior" crawler (and not just a guy with a 4TB disk) is at the mercy of the website's remaining staff and management. Just ask Soundcloud. Archive Team started to archive their stuff when it looked like they were going to go under, but Soundcloud shut them down.