What if Archive.today published torrents of its datasets?

Clickable links:

[0] Archive.today: on the trail of mysterious guerrilla archivists of the Internet - https://news.ycombinator.com/item?id=37009598 - Aug 2023 (58 comments)

[1] "archive.today is doomed to die" - https://blog.archive.today/post/657822115776610304/not-respe...

>It's estimated the quantity of data under management is 1 petabyte. Given they've been going for around 8-10 years, we can extrapolate this is only 100-150 terabytes per year.

You're assuming the volume has been flat for the past decade. Not exponential or even linear growth. That doesn't pass the sniff test.

> What if the developer published the data to a torrent on a monthly or other periodic basis? While not completely legal, it would enable their legacy to live on indefinitely and ensure future access to a lot of information

No, it isn't a solution in my opinion, torrents need to be seeded, (at least) initially by Archive.Today, seeding a shit load of multi terabyte torrents is a lot of bandwidth which will probably up costs

Torrents die, and I would especially expect massive torrents like that to die out fast, there's not a lot of people who can spare that much storage and/or bandwidth, it's not just "go buy a few drives, put in and seed", especially not for that duration of time.

I download and upload more than 50TB per month, every month, just for my personal stuff.

Bandwidth is dirt cheap these days.

> Bandwidth is dirt cheap these days

Not at scale. 50TB per month is nothing. That’s 151mbps. Most transit providers will be okay with a 10% commitment, so this is a gigabit port which is like $60-300/mo.

Assuming 50 people would download the 1PB archive from the provider, that would require at least 151gbps of bandwidth, rounded up to 200gbps, at the cheapest transit provider (HE) this is still at least $12k/mo for transit alone.

Is that dirt-cheap for a non-profit? I’m not sure.

Torrents make distribution even more dirt cheap. See sibling comment: https://news.ycombinator.com/item?id=37028576

In theory yes, but we’re talking about such a massive dataset that archive themselves would have to seed it forever anyway. But yes, in theory, if there would be a lot of seeders for it, this would be a non-issue, however I highly doubt that there are enough people interested in seeding a petabyte worth of archives. It’s certainly less than people who are willing and able to keep a local copy.

And if you did that with a consumer ISP, they would terminate you as a customer very quickly. At which point you’ll have to pay cloud provider for traffic, or a T1 network for IP transit (alongside colocation fees and actually putting a server in a datacenter and all that).

I'm already doing more than 1.5PB/mo, and my ISP doesn't care. I also don't think they'd care if I did 10PB a month, in fact they want to sell me 5Gigabit internet which could do 50PB/mo.

I also have several other unmetered gigabit links. The bandwidth is out there.

This is definitely an exception, not the norm.

> At which point you’ll have to pay cloud provider for traffic, or a T1 network for IP transit (alongside colocation fees and actually putting a server in a datacenter and all that).

And there's an issue when you consider not all content in the archive is legal (both in terms of copyright and other types of illegal), cloud providers, a lot of other hosts will probably kick you off.

Publishing even just the URL table would be a much smaller quantity of data and yet of immense archival value.

> this is only 5 or 6 hard drives at today's maximum capacity of 28TB

And 500 IOPS.

It's fine for the personal usage, it's another thing to seed this amount

If you released a monthly snapshot it'd be ~10TB, which is still a lot but definitely feasible.

If it gets packaged as a series of daily tar archives the IOPS won't be a significant issue. Hard disks perform well in sequential reads.

The beauty of torrents is that as soon as someone else starts to download, provided they're a good swarm member, the overall seeder capacity increases.

You don't need to have 100% of the data to start seeding, you only need a single chunk.

It might take several weeks for that first peer to hit 100% but it should be quickly followed by a whole lot more as those peers are all seeding each other up to 99% amid the process.

Thanks, my first torrent was published on TPB in 2005, I know how that works.

> It might take several weeks for that first peer to hit 100%

And this is a problem here. More leechers -> more IO for the random blocks -> less overall IOPS.

If you find a torrent client what can advertise the blocks sequentially then it would be much faster.

I think the problem you're talking about is solved by super-seeding.

https://en.wikipedia.org/wiki/Super-seeding

I believe libtorrent supports it, among others.