Hacker News Reader

Comment by drewcrawford | Hacker News Reader

The system you've described doesn't really scale to, say, 1000 people with 10TB of files. Even with a lot of flash drives in that cache.

The sneakernet is engineered basically to let you keep making trips to that cache outside your dorm room and somehow magical files appear from a block or two away. Also, files from you magically appear a block or two away when other people ask for them. The sneakernet takes care of all the routing to make lots of files show up in lots of places geographically close to you without too much overhead.

I've found lots of times I will request a file that the system in its cleverness has already cached locally on my HD. So "getting it off of the sneakernet" is actually instantaneous.

View on HN

> I've found lots of times I will request a file that the system in its cleverness has already cached locally on my HD. So "getting it off of the sneakernet" is actually instantaneous.

Huh? I thought that the point was that request files and then get them. How do you already have a file that you didn't request? Is this distributing files across people (i.e. FreeNet) so you have a bunch of files that are only on your machine for the purpose of being an intermediary to others? Or is this running some sort of predictive algorithm that tries to figure out what you will request before you do?

I think these files were on the disk for the purpose of distributing them to other who had requested them.

In a very generic answer to your question, suppose you're moving a 1GB file from A to B with an 8GB flash drive. It doesn't cost anyone anything to move an extra 7GB of data that somebody might need later. You can 8x your throughput for free.

But, at the same time, you don't want a flash drive to be in a constant state of being near-full so it can't fit the new StarCraft beta.

So there's some logic that takes files that seem to be spreading around anyway and sort of spreads them around ahead of when they're requested. This both increases redundancy and increases network throughput. It's not exactly machine learning, but it does, in practice, provide decent results.

By "magic" do you mean "through the internet" or is something cleverer going on?

Also, by cache are you generally referring to some place where files are stored? Or does it mean something else?

I understand the system I just described doesn't really scale well, I'm just not really understanding how it does work from the documentation in the wiki.

cache == a flash drive or group of flash drives (although it could also be a USB HD, etc.)

Not through the internet. Through moving from flash drive to flash drive. If you follow the "testing" section along with the network maps for both the "basic configuration guide" and the "large configuration guide", that should clear up a lot of the magic.

Oh ok, now it makes sense.

You might want to say like storage device instead of cache, that would have made it less confusing for me to understand from reading the wiki.

It also seems like the system is just begging for caches to go missing (though I understand that as all the files are encrypted, it should be hard in theory to actually get data if you arent authorized access to sneakernet).

Are you hashing the files all the time to make sure that files aren't getting corrupted in transit or when taken from someones external cache?

Reply on news.ycombinator.com