Show HN: Synapse, a full featured BitTorrent client in Rust | Hacker News Reader

Show HN: Synapse, a full featured BitTorrent client in Rust

github.com

195 pointsLuminarys71 comments

Not that I'm not always eager to see something get rewritten in Rust, but I'm curious whether web browsers have advanced to the point where a bittorrent client could exist as a webapp rather than requiring native installation. Could the bittorrent protocol operate over WebRTC?

http://webtorrent.io

Does it seriously download a 129MB file just by opening the page?

Ha yeah. Everyone pointed that out the first time this came up and I guess he ignored the feedback.

I actually find it seriously impressive. It demonstrates it works for you right now, and it works seamlessly!

It is impressive, and in a sense it's a very... effective... demo. The problem is that not all of us are in a position to spend that much bandwidth by accident.

It's impressive, but it doesn't necessarily actually download the file from webtorrent peers. It will happily do HTTP streaming, instead.

It works fine for me even though my browser is configured to block third party traffic by default.

Ah, it's a bit misleading then

chunk by chunk, and it's kinda the purpose, maybe they should have a "ready ? go" button ..

Synapse is not designed for people who want this use-case. It's primarily designed for seedbox users who want a daemon to control remotely. It's also designed to efficiently work at scale, and takes advantage of native features like mmap to download quickly.

Also, it wasn't rewritten in Rust from some other language, it was written in Rust in the first place.

Is a seedbox basically a torrent-focused media server? Probably worth submitting a PR to note this focus in the readme. And yes, I'm joking about the rewritten-in-Rust part. :)

>Is a seedbox basically a torrent-focused media server?

No, it's just a box dedicated to "seeding", a server whose purpose is bittorrent.

To clarify further, a seedbox is a server used to continually "seed" (or share) any content downloaded to it.

Seedboxes are mostly used on private torrent trackers, simply because the majority require users to maintain a high seed/leech ratio. In other words, users need to upload more than they download.

How does mmap improve download speed?

Not the author, but I think that the point is to make disk I/O async with respect to requesting and receiving blocks. I.e. request for the next block does not wait for the write of the previous block to complete.

And this makes sense when disk is slow and you have much more RAM than network traffic. If RAM is small, while net traffic and disk are both fast, mmap writes may result in poor file layout due to arbitrary OS flush order.

Is it also to reduce security concerns?

I chose Rust because of its combination of ergonomics and low level control available. The safety came as a bonus.

Ergonomics? I love rust as much as the next guy but ergonomics was the one recurring pain point for me. So much of the language is inconsistent/incomplete and the debuggers barely work sometimes.

When was this? We've been putting a lot of work into ergonomics. We're not done yet; more stuff is coming. But depending on when you last used it, that may explain the discrepancy.

Or maybe not.

> inconsistent/incomplete

I'd be interested in hearing more about this, if you have the time. Rust is certainly incomplete, but I wonder if our perspectives line up here or not regarding how exactly it's not complete.

I wrote a few hundred lines about bout a year or two ago and about 2k the last few months. It's definately gotten better and there's something in the works for most of the things I took issue with which is kind of nice but doesn't really help if you're not on nightly.

Specific pain points for me were: - the lack of traits for numeric primitives/types without pulling in the num crate - the difficulty working with floating point types - the inconsistent type annotations in fn signatures - the type annotations are a bit clunky no easy way to alias - and auto deref mechanics especially via match statement - having to fully specify a type in a match statement or use Self::Type - the use mechanism makes no sense - no optional/default arguments without macros leads to a lot of shimming/boilerplate - having to split operations because the borrow checker can't do partials or non lexical lifetimes - the clunky range syntax wich makes writing invertible/reversible functions really ugly and error prone due to lack of inclusive range and I really can't believe the committee decided to go ahead with ..= instead of ... It's really out of line compared to pretty much every other language is doing - doesn't seem like there's a good way to nicely format long lines

Overall I still really like the language but I really don't see it taking off until these issues are fixed because it makes it so hard to start up.

Cool, thanks. Some of these things are being fixed, some of them, not, or at least not any time soon. (FWIW, I agree with you regarding =. We'll see...) I could go through the exact details if you care, but I won't bore you with it otherwise. Thanks for the feedback; it's really helpful.

What kind of rust experience do you have before embarking on this project? Also, did you do systems programming before?

The Opera web browser had a torrent client built-in for a long time.

https://en.wikipedia.org/wiki/History_of_the_Opera_web_brows...

Looks like since version 9, released in 2006.

Honestly it should be standard in all browsers.

What if I'd argue otherwise - that download managers (including plain HTTP(S) downloads) shouldn't be a part of browser, but a separate pieces of software?

I suspect that in modern "we don't download anything, we watch/read/run this on the web" times in-browser downloading is bound to be second-class citizen that vendors would not pay any significant attention to beyond the simplest scenarios. Basically, the torrent support will be barely functional, just like the current HTTP downloads are. If they're split in a separate program (buzzword: microservice) they still have a chance, though.

What if I'd argue otherwise - that download managers (including plain HTTP(S) downloads) shouldn't be a part of browser, but a separate pieces of software?

Well, they used to be, but now they aren't anymore, so you'll probably want to start somewhere around there. In the early days of resumable HTTP downloads and simpler browsers, there were lots of download managers.

https://www.internetdownloadmanager.com/support/using_idm/us...

That's pretty much WebTorrent, isn't it?

So, if I understand this correctly, Rust would be a good fit for bittorrent, right? Handling lots of stuff simultaneously, peers, trackers etc, to me a bittorrent that actually does more than download and quit always seems like something quite heavy...

>a bittorrent that actually does more than download and quit always seems like something quite heavy...

Don't forget to seed!

Let's brainstorm ways to solve the seeders/leechers problem via cryptocurrency: how much would it cost to encode a complete feature-length film in the bitcoin blockchain? :P

The user would torrent a scrambled file, once some conditions were met the user would receive the mechanism for unscrambling the file. While the user is seeding the scrambled file they would have an alternate metadata that describes that file. The version that is served could be based on consensus of the network.

Bram Cohen built MojoNation which was the first real cryptotoken/smartcontract startup

After that failed he designed BitTorrent to actually align the incentives of users, without any of the currency bullshit getting in the way

a solution already exists. it's called a private tracker.

Seems like having every single full node storing multi-gigabyte files for the few who actually want them wouldn't go over too well.

Rust would be a good fit, yes, but many other languages could be a more comfortable fit. For example since we're just doing a simple data processing task, a simple language like Go would work perfectly. Since the realtime constraints are not super strict, you can build fine BT clients in comfortable languages such as C# or Java as well.

An advantage of Rust however is that it's easy to expose a C api from it, so it can easily be integrated into a project done in a more comfortable language.

It does seem like a good fit, yes. Just for example, rtorrent regularly crashes every few days.

Security is the main gain for many people.

For automation software, transmission (BitTorrent-client written in python, slow) seems to be the standard.

If it could offer a transmission-compatible API-layer, I bet it would receive much more interest.

Edit: apparently not written in python. My bad.

Your wish is satisfied: https://github.com/SirCmpwn/broca

Nice. Appreciated. Now I clearly have a weekend project.

(Also, what great irony this proxy is written in python, eh?)

>(Also, what great irony this proxy is written in python, eh?)

It's written by a different person (me). Also Transmission is written in C FYI

Yes. That was what was ironic since I (wrongly) complained about slow python based software.

The first implementation of BitTorrent was written in Python. I don't think it has happened once since, not as any serious attempt anyway.

The clients that use Python today are not slow because they all use rasterbar-libtorrent as a C Python module that performs all of the heavy lifting.

What do I use it for? The README isn't very explanatory.

Transmission is written in c, not python.

It would be great to have a client that works well for seeding massive amounts of torrents (100,000+) on weak hardware (like a Raspberry). The main problem seems to be memory usage.

How is the memory usage of this one?

It says it is light-weight. What kind of light-weight is it and are there some metrics?

That's an interesting use case you have there... care to explain the motivation?

Wanting to make sure that lots of rare things stay accessible. ;)

Even if you solve the memory issues you will still be constrained by the NIC.

Most of these SBC's run ethernet over USB, so the performance is still going to be shit.

Probably still good enough to saturate the typical American broadband connection though.

I recently selected an ODROID-C2 for a NAS because of its dedicated Ethernet controller. Cute device. My fingers are crossed for the performance test early this week.

If you want to just make sure that lots of content are accessible eventually that is neglible (and I am constrained by ISP anyways).

Please help test it! We currently mean lightweight in terms of design, but we also want it to be lightweight in terms of performance. But we haven't had many opportunities to test it with truly huge numbers of torrents (though we have tested it on large torrents (1.5T+) with lots of files (1000+)).

As far as memory usage goes, I don't think low-memory devices have been a use-case Luminarys has had in mind yet, but he's probably on board with tuning with that in mind. On my client right now I have 45 torrents, 3 seeding, and one is that huge one I mentioned before - and synapse is using 1.2G of RAM, and 41G of virtual memory.

That's a lot of memory...my little NAS has 512mb IIRC.

I run Transmission (from the Debian repo) on it and never even thought about memory usage before now, it just works. Admittedly I don't really let a bunch of torrents run at once though.

Yeah. I've opened a ticket to audit memory usage:

https://github.com/Luminarys/synapse/issues/80

I should note that I'm stress testing it, however, I have some particularly large torrents going.

Awesome work, congrats!

Unfortunate name clash (https://github.com/matrix-org/synapse)

Someone ought to write a website that scrapes popular OSS hosting sites and package repositories and comes up with a list of English words that haven't been taken already. :P

waiting for kibwen to post a Show HN

Make it, and then name this app "Dictionary" :P

More name clashes: https://en.wikipedia.org/wiki/Synapse_(disambiguation)

Anyone using a common English word shouldn't expect to own it exclusively, especially if it's a word that has something to do with computation.

Proposal: Rusty Sintorrent

Interesting ! I tried integrating transmission for downloading content in a video wall app and it was a bit of a nightmare.

Hopefully this will be better - please have a sane remote API.

One thing that Azeurus did really well, that transmission really did not -

You could can tell azeurus to re-check all the files that are there so it can find out how far it's got.

In Azeurus I used this to reconstruct a large torrent by adding files that I found, then telling it to re-check.

Transmission couldn't even tell if the files the torrent pointed at had been deleted from under it.

Really? That's odd, that's a pretty standard feature in every client I've tried. I'm not positive, but I'm fairly sure transmission does it too.

>please have a sane remote API

https://github.com/Luminarys/synapse/blob/master/doc/RPC

Pretty much all torrent clients can recheck the files.

Transmission does this with the verify command.

The python API for transmission is pretty simple, I wrote a RSS feed reader that adds new torrents in something like 20 lines of code (not counting my custom rss parser module automagically generated from the schema).

> in Rust

Saying what language a program uses only describes its grammar and runtime. Saying what dialect a program uses also describes its idioms.

https://en.wikipedia.org/wiki/Programming_idiom

For (a simple) example, actix-web uses actix, a library that largely changes the nature of writing Rust code. This might be called Rust-a or Rust', or something.

https://github.com/actix/actix-web

My point is that Rust is a really cool language where "the(/some) (sub)community(ies)" are developing and coalescing around great libraries outside of core std, making a very* robust language, or rather robust language dialect.

I'm not all that familiar with actix, but surely not every Rust program is written using a library that largely changes the nature of Rust code. Like, would you say there's a name for the dialect this program is written in?

Programming dialects don't seem to get names. I think a pattern of [language-name]-[dominant-principles] might make a good name scheme. Suffixes would then work across languages. Other suffix namespaces could coexist, though I'm not sure what they'd be.

My point wasn't so much about any one program, so much as it was about an invisible programming ethnology. I like when Show HNs say what language they speak; I would like even more if they say how they speak it.

Having name for program dialects would allow prospective users and contributors to gain deep insight about its intended structure, vision, scope, limits, etc, etc, etc, etc from a distance.

It may have seemed narrow focused, but I didn't want my point to be limited to one set of principles, patterns, and libraries, even though I like what actix adds to Rust-std(?), especially since any code spoken in it can itself make, or through some elements factored out be made into, other libraries that then build on or toward another powerful dialect / speaker communities.

#hashtag: ethnocodology(?) ..ethnokodikology?

Reply on news.ycombinator.com