Hacker News Reader

Comment by stavros | Hacker News Reader

I see things like this, and I wonder why the following software doesn't exist:

I want a piece of software to which I can add a collection of files, say multiple TB. The software will then behave a bit like a BitTorrent tracker, and know which peer has which files. A peer joining this swarm will be able to say "I want to donate X GB of space", and the tracker would tell it "OK, then download and seed these files, which are the least seeded".

The peer would download the files from the rest of the swarm and make them available to it. Then, a request layer on top of the swarm could be used to request a file from the peer which had it. Adding/removing files to this collection would also need to be a feature.

Does anyone know if anything like this exists? If not, how easy would it be to make something like it out of BitTorrent? I might give it a go.

View on HN

Have always thought this would be a great way for many people to share well organized Plex libraries. Many semi-overlapping libraries basically creating a virtual Netflix, with some way to stream in Plex from the whole library no matter if you actually have the content locally or not

>virtual Netflix

this already exists with addons for Kodi

Freenet is built around similar ideas, combined with encryption an anonymization. Which hopefully adds the benefit of you not being legally liable of distributing CSAM. The gist of Freenet is that you can operate a freesite or upload files, and the content is redundantly dispersed in encrypted parts among a number of other Freenet peers. They don't know what they're hosting and neither do you, the client just fills up the allotted space and uses some bandwidth, that's all.

https://en.wikipedia.org/wiki/Freenet

Regarding of how easy to make something like this network, I'd wager it's pretty hard. There will be a lot of questions, even while establishing the happy path, for example how you manage the updates, especially when you update the protocol, not just the software, and how you effectively manage the volume of search requests, how you distribute the files etc.

And then there's the abuse the network will inevitably get. How you handle spammers, CSAM, malware, ISPs that throttle/block you, the legal risk you put your clients up to, etc. Nice big can of worms. To begin opening it, I suggest a reading through Wikipedia's Peer to peer file sharing article, and especially the File sharing modal on the right, which nicely captures the ideas that have been tried so far.

https://en.wikipedia.org/wiki/Peer-to-peer_file_sharing

My idea is similar but slightly different. You'd run your client and then choose whose provider's data to seed, eg you'd add the Internet Archive and Libgen to your datasets.

Only those entities would be able to push data to you, nobody else, so if you trust the providers you specified, you should be good.

I remembered that I read about something like that before. I think this was it: https://en.wikipedia.org/wiki/Storage@home

Further this lead me to distributed data stores: https://en.wikipedia.org/wiki/Distributed_data_store#Peer_ne...

That's interesting, I'll read up on it, thanks!

I think BitTorrent has all the pieces needed for a fully distributed version of your idea. My initial thought is that you could publish a magnet link that points to a mutable DHT item, which in turn points to a torrent that has a JSON file with some metadata and a list of infohashes the publisher cares about. The client could then scrape the "leaf" torrents from multiple lists to get the peer counts and use that for local prioritization of what to store. By reusing existing torrents you could then share resources with standard torrent clients that are unaware of your system.

The list idea could be extended to nested lists (stavros recommends Internet Archive) for discoverability and composition.

If you go with v2 or hybrid torrents from the beginning you could deduplicate and cross seed files from different collections.

The lists could also be modified to have torrents to exclude, possibly using some salt + rehash idea to make it hard to reverse into a list of e.g. CSAM you don't want to publish as is.

Feels like a neat project that could interoperate nicely with existing torrents.

Thanks, that's exactly the feedback I was looking for! This sounds like it would work, though I'd have to see if it would scale to thousands or millions of files. Still, great for a PoC, thank you!

Donating space is only half of the equation. I think donating bandwidth is a more significant aspect, especially with ISP's like Comcast which provide very little upload bandwidth compared to download. You'd expect that uploads wouldn't impact download speeds, but it's not the case. A saturated upload bandwidth means, ACK packets getting delayed, which means connections would be established way more slower. So, it's not a feasible prospect unless the competition takes over.

You can always implement QoS or something like FQ_Codel.

I'm a senior software developer with experience in developing communications protocols and networking software. I haven't been able to set up QoS properly ever, especially in a way that addresses all my needs. Expecting end users to do it is way beyond the realm of possibility IMHO.

It is a huge pain in the ass to get it working right I will admit. Took me hours of reading docs and tweaking parameters. I'm still not sure I understand all of it, but I managed to get it working such that it at least meets my needs.

Something similar exists: iabackup[1][2]. It is designed to host an independent copy of (some of) the Internet Archive using git-annex. You tell it how much storage you want to donate and git-annex fills your disk with data from the least-seeded files IIRC. Its focus is on data backup, not data serving though.

[1]: https://git-annex.branchable.com/design/iabackup/ [2]: https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/g...

That's literally how most of Japanese P2P software work.

For example, Perfect Dark, Winny, and to less extent, Share (which is more similar to eDonkey/eMule).

It's proprietary, but it sounds like you're describing the "Google File System". https://en.m.wikipedia.org/wiki/Google_File_System

For your idea, once all the local storage everywhere is filled up with evenly distributed redundant copies, and then a new file is added, would peers arbitrarly choose other files to delete in order to make room for the new file?

Yes, it's very similar, though would work slightly differently (BitTorrent under the hood, whole files instead of chunks, etc).

This sounds very similar to ipfs. How does it differ?

IPFS can't give you the least-seeded files. It can't give you any files, you have to manually pin the set you want, and you pin all of it, and it won't automatically change.

It's not very convenient for archival, whereas the system I'm talking about would be (in my opinion, anyway).

(How) would you filter junk?

You'd trust the publisher, so you'd say "I want to help the Internet Archive with its archiving" and they'd be responsible for what files they pushed to your storage.

This is basically an opportunistic, distributed filesystem that's designed to work with high latency links. Only the tracker can write to its nodes, but anyone can read.

This, and easy-to-configure bandwidth throttling was sorely missing from IPFS last time I tried it.

Being able to specify "use 50% of storage to cache low-traffic things, and the other 50% for high-traffic things" would be amazing.

Not part of the question or problem statement as I see it.

Private trackers do this somewhat via incentives. Files with less seeds earn you more bonus points. If you want to farm BP you can sort files by seeders and download poorly seeded torrents.

I keep thinking in my mind this makes sense but then ponder about the "local" impact - what would a user "donating space" be storing locally and might risk? Where does the risk lay, is it with the uploader, the seeder, or the service provider? I would love to see this pickup but it feels as though it would devolve into one of the listed projects. It "sounds" as though it should use some kind of Distributed Ledger Technology, IPFS seem the most fit, maybe some kind of next layer that offers just this service (IPFS being GDPR compliant might mitigate some risk?)

Reply on news.ycombinator.com