HNHacker News
TopNewBestAskShowJobs

rklaehn

1,071 karma · joined October 16, 2017

submissionscomments
rklaehn··on Iroh: A library to establish direct connection between peers
The two sides are peers when it comes to connection establishment, but once you have a connection they can and frequently will have different roles.

Many existing iroh protocols have clear client and server roles once the connection is established. E.g. gossip is a peer to peer protocol, blobs is a client server protocol in that one side provides data and the other requests it.

For a client you can use an ephemeral node id and not publish your info to discovery, since you will never be dialed yourself.

rklaehn··on Iroh: A library to establish direct connection between peers
We decided to keep things simple. In general we try to provide one good way to do something instead of having a lot of options.

E.g. we only provide Ed25519 for keys and in iroh-blobs only BLAKE3 for hashing, instead of having a multihash scheme. Having the public key directly available is sometimes useful, e.g. for verifying signatures. It also allowed us to directly use the mainline extension BEP_0044 to store data for public keys.

That being said, I am very confident that we will be able to provide a relatively smooth transition if we ever have to switch from Ed25519 to another public key format.

For connection encryption we use a TLS extension called raw public keys in TLS, and here of course the keys are prefixed, and we could easily upgrade to another key format and then at some point stop supporting Ed25519 keys.

raw public keys in TLS: https://datatracker.ietf.org/doc/html/rfc7250 storing arbitrary data in the DHT: https://www.bittorrent.org/beps/bep_0044.html

rklaehn··on Iroh: A library to establish direct connection between peers
Yes. Iroh itself provides direct QUIC connections. iroh-blobs is a protocol on top of iroh that provides content-addressed data transfer of BLAKE3 hashed data.

What you describe sounds like https://www.dumbpipe.dev/ , a tool/demo built on top of iroh to provide a bidirectional pipe across devices, somewhat like netcat.

Dumbpipe also has a mode where it listens on a port using TCP.

It sounds like you want to basically build dumbpipe for UDP. You can of course use a QUIC stream, but QUIC has an extension, which we support, to send datagrams: https://docs.rs/iroh/latest/iroh/endpoint/struct.Connection....

This basically allows you to opt out of QUIC streams, but you still do get TLS encryption.

rklaehn··on Iroh: A library to establish direct connection between peers
To expand on this: iroh is a rust library. A NodeId is just an Ed25519 public key, but of course it has a distinct type. If in the future we want a different public key standard, it would be a different rust type.

Encoding keys is mostly left to the user. The only exception are tickets. Tickets are postcard serialized and have a version field, so we can keep tickets compatible if we ever want to use a different public key standard or hash function.

(disclaimer, I also work on iroh)

rklaehn··on Ask HN: Why is there no P2P streaming protocol like BitTorrent?
OK, thanks for the clarification.

I have nothing against crypto in principle, but I really don't want Iroh to be associated with crypto scams.

Iroh is just a library for p2p connections. You can use it for crypto, but I would say that the majority of our users are non-crypto(currency).

We will try to make the wasm version easier to use, but if nostr works well for you, go for it! Not the right place if you want to avoid crypto bros though :-)

rklaehn··on Ask HN: Why is there no P2P streaming protocol like BitTorrent?
I am a contributor to Iroh ( https://github.com/n0-computer/iroh ), an open source library for direct QUIC connections between devices that can be behind a NAT.

Our library is general purpose and can be used whenever you need direct connections, but on top of Iroh we also provide iroh-blobs, which provides BLAKE3 verified streaming over our QUIC connections.

Blobs currently is a library that provides low level primitives and point to point streaming (see e.g. https://www.iroh.computer/sendme as an example/demo )

We are currently working on extending blobs to also allow easy concurrent downloading from multiple providers. We will also provide pluggable content discovery mechanisms as well as a lightweight content tracker implementation.

There is an experimental tracker here: https://github.com/n0-computer/iroh-experiments/tree/main/co...

Due to the properties of the BLAKE3 tree hash you can start sharing content even before you have completely downloaded it, so blobs is very well suited to the use case described above.

We already did a few explorations regarding media streaming over iroh connections, see for example https://www.youtube.com/watch?v=K3qqyu1mmGQ .

The big advantage of iroh over bittorrent is that content can be shared efficiently from even behind routers that don't allow manual or automatic port mapping, such as many carrier grade NAT setups.

Another advantage that BLAKE3 has over the bittorrent protocol is that content is verified incrementally. If somebody sends you wrong data you will notice after at most ~16 KiB. Bittorrent has something similar in the form of piece hashes, but those are more coarse grained. Also, BLAKE3 is extremely fast due to a very SIMD friendly design.

We are big fans of bittorrent and actually use parts of bittorrent, the mainline DHT, for our node discovery.

Here is a talk from last year explaining how iroh works in detail: https://www.youtube.com/watch?v=uj-7Y_7p4Dg , also briefly covering the blobs protocol.

rklaehn··on Ask HN: Why is there no P2P streaming protocol like BitTorrent?
> Shame it's being abused by crypto bros who want to treat it as money.

Iroh contributor here. I don't know what you are referring to. Iroh is just a library to provide direct QUIC connections between devices, even if they are behind a NAT. We don't have any plans doing a blockchain or an ICO or anything like that.

I am not aware of any project called Iroh that is a scam, but if there is, please provide a link here. It's not us.

I know there have been some scammers trying to make a BLAKE3 coin or something, a year ago.

rklaehn··on Pkdns: DNS server resolving via mainline DHT
We use pkarr at iroh.computer for node discovery. It is anything but slow. It is very rare for a lookup to take more than a few milliseconds. It is sometimes faster than our non-p2p node discovery option which is using DNS.

DHTs get a bad rap because of many recent DHTs that were horribly inefficient. But mainline is different. Many of the design decisions of mainline seem very limiting at first, but make a lot of sense for perf.

E.g. a pkarr record can only be 1000 bytes, so the entire message fits into a single non-fragmented UDP packet.

rklaehn··on Public-Key Addressable Resource Records
The simplest possible integration between DNS and peer-to-peer overlay networks
rklaehn··on Willow Protocol
That is not a problem specific to IPNS though. Using a DHT for something like IPNS is fine. Publishing roots of large data sets is also fine(ish).

Using it to publish every tiny chunk of a large file is a horrible idea. It leads to overwhelming traffic.

If you publish a few TB of data, due to the randomness of the DHT xor metric you have to basically talk to every node on the network. Add to that the fact that establishing a tcp libp2p connection is much more heavyweight than sending a single UDP packet like in the bittorrent mainline DHT, and you are basically screwed.

In iroh we don't publish at all by default. But if you have to use a DHT, the fact that we have a single hash for arbitrary large files due to blake3 verified streaming helps a lot.

You still get verified range access.

rklaehn··on Willow Protocol
Not yet. We have been busy with other stuff, also the willow spec has been a bit of a moving target until now.

We would like to take our rust willow impl and separate it a bit more from our code base, so that iroh documents are just users of the willow crate.

rklaehn··on Willow Protocol
Thanks for letting us know. We will add a section about encryption.

These tiny tools are basically one week projects to show off the tech, but they try to be useful on it's own as well.

rklaehn··on Willow Protocol
Difficult to answer.

IPNS uses the IPFS kademlia DHT, which has some performance problems that you can argue are fundamental.

For solving a similar problem with iroh, we would use the bittorrent mainline DHT, which is the largest DHT in existence and has stood the test of time - it still exists despite lots of powerful entities wanting it to go away.

It also generally has very good performance and a very minimalist design.

There is a rust crate to interact with the mainline DHT, https://crates.io/crates/mainline , and a more high level idea to use DHTs as a kind of p2p DNS, https://github.com/nuhvi/pkarr

rklaehn··on Willow Protocol
So, I am not the one that is doing the bindings, so take this with a grain of salt.

It seems uniffi does create C compatible bindings in order to make bindings for all these other languages. But these are internal bindings that are ugly and not intended to be used externally.

rklaehn··on Willow Protocol
It's using the same transport. Basically sendme is like dumbpipe, but adds blake3 verified streaming from the iroh-bytes crate for content-addressed data transport.
rklaehn··on Willow Protocol
To be fair, IPFS does offer not just content addressing but also a mechanism for mutability with IPNS. You can think of a willow namespace (or iroh document) as a key value store of IPNS entries.

The problem with IPNS is that the performance is... not great... to put it politely, so it is not really an useful primitive to build mutability.

You end up building your own thing using gossip, at which point you are not really getting a giant benefit anymore.

rklaehn··on Willow Protocol
Thanks. This is awesome. I think they are doing more work themselves in terms of crypto, whereas we rely on QUIC+TLS more.

Regarding holepunching, our approach is a bit less pure p2p, but has quite good success rates. We copy the DERP protocol from tailscale.

I am confident that we have a better story regarding handling of large blobs. We don't just use blake3, but blake3 verified streaming to allow for range requests.

Also I wrote my own rust library for blake3 verified streaming that reduces the overhead of the verification data. https://crates.io/crates/bao-tree

I tried to get on their discord at https://veilid.com/discord, but I get an invalid invite. You know a better way to get in touch?

rklaehn··on Willow Protocol
This looks very interesting. They made very similar choices than we (iroh) did. Rust, ed keys, blake3.

They seem to do their own streams, while we are adapting QUIC to a more p2p approach. Also the holepunching approach seems to be different. But I would love to get more details.

rklaehn··on Willow Protocol
It's more generic than that.

Syncthing is designed specifically for file system sync (and does a very good job). Willow could be used for file system tasks, but also for storing app data that is unrelated to file systems, like a KV store database.

You should be able to write a good syncthing like app using the willow protocol, especially if you choose blake3 as the hash function.

rklaehn··on Willow Protocol
This is a good question. But it is worth noting that not everything has to scale globally.

E.g. in iroh-sync (which is an experimental impl of the willow protocol) you are not concerned with global scaling. You care only about nodes that are in the same document.

So while if you request hash QmciUVE1BqKPXMSvTTGwHZo1ywYdZRm9FfBvEJkB6J4USb via ipfs, you are trying to globally find anybody that has this hash, which is a very difficult task.

If you ask for some content-addressed data in an iroh document, you know to only ask nodes that participate in this particular document, which makes the task much easier.

Edit: regarding clients, iroh is released for osx, windows and linux. Iroh as a library also works on ios. Download instructions are here: https://iroh.computer/docs/install

rklaehn··on Willow Protocol
Prefix pruning is a very different approach than tombstones. An update will actually remove data, and not just mark the data as removed.

Maybe "total erasure of data" is a too strong promise, but the fact that you can not force nodes that you don't control to unsee things is common knowledge, so in my opinion this does not need a qualifier.

rklaehn··on Willow Protocol
Not officially. We currently have bindings for rust, python, golang and swift.

These were the most asked for bindings (python for ml, golang for networking and swift for ios apps).

We are using uniffi https://mozilla.github.io/uniffi-rs/

Would you need C or C++ bindings?

rklaehn··on Willow Protocol
Think of it like const generics in languages like rust or C++.

You can make two data structures with a const parameter.

If the parameter is not the same, they are not compatible (not the same type). The parameter can be tuned according to the specific needs of the application.

rklaehn··on Willow Protocol
I was not aware of iRODS.

Iroh is named by a certain fictional character that likes tea. Any similarity is a coincidence.

But it seems like iRODS is much more high level than iroh. E.g. iroh certainly does not contain anything for workflow automation. You could probably implement something like iRODS using iroh-net and iroh-bytes.

rklaehn··on Willow Protocol
if you are looking for something similar to ipfs but a bit more minimalistic and performance oriented, check out iroh https://github.com/n0-computer/iroh .

It is a set of open source libraries for peer to peer networking and content-addressed storage. It is written in rust, but we have bindings to many languages.

One part of iroh is a work in progress implementation of the willow spec. The lower layers include a networking library similar to libp2p and a library for content-addressed storage and replication based on blake3 verified streaming.

Most iroh developers have been active in the ipfs community for many years and have shared similar frustrations... See this talk from me in 2019 :-)

https://youtu.be/Qzu0xtCT-R0?t=169

rklaehn··on Willow Protocol
iroh developer here.

willow was not developed in a vacuum.

the willow folks have worked with us while we have implemented many ideas from willow, starting with range based set reconciliation ( https://arxiv.org/abs/2212.13567 )

they have been open to removing parts that have turned out to add too much complexity to implementations.

rklaehn··on What Will the World Look Like with Less Babies?
Yes, this is very true. If you listen to some people you would think that once you have kids your life is basically over. You will never be able to spend any time on yourself anymore.

This is just not true - at least if you refuse to become an extreme helicopter parent.

I have four kids. I had a ~10 year break where I very rarely made it to parties and music festivals, and my life was mostly work and kids. I had a great time and have never regretted my decision to have kids, but there were some things I missed.

Now we start doing these things again, sometimes with the kids, sometimes because the kids are big enough to be alone or because we can afford a babysitter.

Here is one egoistic argument for having kids: the world is going to become very confusing in a decade or two. There will come the time where you can't keep up anymore. Then you will be happy to have somebody that loves you unconditionally that can make decisions for you. I trust my kids way more to take care of me when I am old and confused than any institution or person motivated by money.

rklaehn··on What Will the World Look Like with Less Babies?
> Your comment has the whiff of Magical Tech thinking to it.

Well, your post has the whiff of malthusian doomerism to it. Look up the Simon-Ehrlich wager. Ehrlich was wrong in every single respect. There are more people, yet they are economically better off. And nothing has run out.

And yet the ideology of Ehrlich and Malthus has prevailed in the west.

https://en.wikipedia.org/wiki/Simon%E2%80%93Ehrlich_wager

rklaehn··on What Will the World Look Like with Less Babies?
For every single decade in the 20th century there was some kind of doom hanging over our head.

Imagine deciding to have kids in the 80s where the two superpowers were pointing insane amounts of nuclear weapons at each oter. Or in the 70s when people feared a new ice age. Not to mention 1910 or 1930.

You may say "but this time it is real". The nuclear threat in the 80s was very real as well. As was WW1 and WW2.

Finland has done it's part against climate change. They built a large nuclear power plant, and can run the rest of the economy using PV and wind. Retire ICE vehicles, and then be proud of yourself and be more optimistic.

Even in the worst case scenarios the climate in finland will be just fine...

rklaehn··on What Will the World Look Like with Less Babies?
Why? There are huge stretches of the world that are basically uninhabited.

Even in densely populated places like Europe there are large areas with low population density.

People are crowding in cities and then complaining that there are too many people.

Resource drain is no longer an issue once we got an economy running on photovoltaics and nuclear energy. Everything can be recycled given enough energy.

← PreviousPage 4 of 10Next →