I've been an editor on Wikipedia for years, and it's simply amazing how many web pages I referenced in 2008–09 have disappeared. Digital archivists have their hands full.
There was an article by Brewster about preserving the Internet in the scientific American which put the average lifespan of a URL at around 40 days. The said article now 404s and the only way to get it is through the wayback machine. :)
http://web.archive.org/web/19970504212157/http://www.sciam.c...
I worked at The Archive for a little while and one of the projects I worked on was to unpack about 300 TB of crawl data from the defunct search company cuil.com. It was mostly decent quality data in a standard format and after some grinding, the whole thing was converted into warc files which the wayback machine could use to show the URLs. The end result was that about 60 billion URLs came "back onto the web".
During the work, I was looking at the stones rather than cathedral but after I left and thought about it in detail, it was very satisfying. I was reading the book "A Canticle For Leibowitz" at the time and the general theme of cycles of history was in my head. That dovetailed very well with the work I had done.
If you're interested, you can download the dumps over here https://archive.org/details/cuilcrawl
As for the book, I think if The Archive had novel for a totem, it'd be "A Canticle For Leibowitz". Very much affected my world view when I read it and this thread has just kindled my interest again. :)
One of the more curious cases was CSIRO (Australia's national science and research organisation) which seems to have not only deliberately purged a fair amount of data (Graham Turner's work specifically), but has a robots.txt in place which blocks archival by TIA. That strikes me as ... downright curious.
I'm thinking that could be something to have an Opposition Senator bring up in Senate Estimates.
404: http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Li...
Now available: http://web.archive.org/web/20120508210658/http://www.csiro.a...
That's among the specific links which wasn't being served by TIA earlier.
dredmorbius@gmail.com if you happen to track that down.