Hacker News Reader

Comment by mnd999 | Hacker News Reader

I think this requires more explanation. Is this some kind of cloud archiving? Are they using ipfs or something like that?

View on HN

Volunteers run Docker containers or Virtualbox VMs at home so that the traffic looks residential and does not get banned. For example: https://imgur.com/a/QXrhudA

Most useful content gets packaged by the ArchiveTeam and send to the Internet Archive (no affiliation between the two).

but it's not really necessary for Reddit. their API is fairly robust and there are numerous options for scraping the site.

AT is not limited to Reddit scraping, take a look at their wiki.

I'm aware of AT and the work they do. Its a great initiative.

I'm just saying that other projects like https://pushshift.io/ have been capturing all reddit posts and comments for years now.

reddit will ipo soon, which is why this is frontpaged now, i presume. folks expect the api to soon become much more restrictive. other mirror projects don't have the ideology or reputation of archiveteam.

I'm also not totally familiar with what's going on but I discovered it on /r/datahoarder and I think it's because redditors are scared that content starts vanishing now that reddit files for IPO.

Anyways, here's a further description: http://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior

Of course it will. Reddit is probably one the largest porn sites.

ArchiveTeam usually sends all the data they collect into Internet Archive. The two are independent, unrelated organizations, but IA is pretty open about accepting hoards of this type.

Reply on news.ycombinator.com