Hacker News Reader

Comment by rklaehn | Hacker News Reader

Congrats for the release. Mostly familiar with IPFS, but learned about DAT at dtn.is last year. The hypercore protocol is pretty well designed and documented.

How would hyperdrive deal with the following scenario: you got a large dataset such as wikipedia. Lots of people have browsed it, but most of them only have a tiny fraction.

How do yo know which peers to connect to to get a particular bit (offset?) you are interested in? The DHT only tells you which nodes participate in the hypercore, not what they have in detail, right?

View on HN

The peers gossip using compressed bitfields in regards to what data they have. These bitfields are super small so we can pack quite a bit of information.

At the moment we don't do anything special in regards to discovery, but as we scale that's something we want to investigate. Since everything is running on append-only logs we can group the data into sections quite easily so there is some easy wins we can do there with announcing to the dht that you have data in a specific region.

So do you opportunistically gather info about what other peers have via this gossip protocol, or just when you need something?

I had looked at https://datprotocol.github.io/how-dat-works/ , but I don't remember anything about a gossip protocol or a peer building a "view of the world". Is that new?

It's only between the peers in your subset of the swarm for now. They exchange a series of WANT and HAVE messages where they subscribe to the sections of each others logs they are interested in.

We are working on expanding this scheme so peers can help discover peers that have the section you are looking for.

Due to the compressed bitfields these section are quite large. In most cases using a few kilobytes you can share WANT/HAVE for millions of blocks

Yes, I saw the compressed bitfields. Most amount of bit twiddling I have ever seen in a pure javascript library...

Being able to identify a piece of content by an integer instead of a hash makes things more efficient compared to content-addressed storage a la IPFS.

Reply on news.ycombinator.com