Show HN: DirtyShare, Pure JavaScript P2P File Sharing (in Node.js + Socket.io) | Hacker News Reader

Show HN: DirtyShare, Pure JavaScript P2P File Sharing (in Node.js + Socket.io)

gun.io

75 pointsMizza25 comments

I made almost exactly the same thing a few months ago: http://www.transfury.com/

The server is written in node.js and it uses socket.io to do the transfers. I think it only works in Chrome because I didn't bother getting the drag-and-drop working with anything else. Actually, it seems to be broken in the current dev channel of Chrome. I announced it back in September:

https://plus.google.com/116396240707733722472/posts/d9ydb8o2...

Interesting! It's funny how we both chose the same arbitrary chunk size :)

Heh. Well, mine wasn't completely arbitrary. If I remember correctly I was trying to trade off server CPU usage with memory usage. One chunk at a time is buffered on the server, so I wanted to keep it small enough that I could have a bunch of ongoing transfers on my puny t1.micro. However, too small of a chunk size resulted in too many roundtrips and lots of CPU load on the server. I settled on this value because I was able to pump a couple megabytes per second through it using one stream, which seemed reasonable.

Also: in further testing, it looks like Safari works with the drag and drop, but not the uploading. I purposely disabled all socket.io transfer types except web sockets in order to encourage adoption of modern browsers. Unfortunately, since I used such bleeding edge, unstandardized technology (drag and drop, file API, and web sockets), the browsers have changed and broken my implementation, each in a slightly different way, such that none of them work end to end anymore.

If anyone cares, I found out what the problem was. Chrome updated to a new version of the web socket standard back in version 15 or so. I wasn't aware of the change and so didn't update my version of socket.io which supports the new standard, thus an incompatibility. It should work now.

This just goes to show that you have to be vigilant if you are building sites with the latest and greatest, as they are still under development and likely to change out from under you.

Couldn't you just connect the uploader's POST to the downloader's GET request instead of using websockets?

This seems to be more of a streaming upload/download. Node just basically pipes along the chunks from the serving peer through the websockets.

I'm pretty sure that if you just "connect the uploader's POST to the downloader's GET" then the Node server would have to buffer the entire file from the submitter before sending it to the clients.

> I'm pretty sure that if you just "connect the uploader's POST to the downloader's GET" then the Node server would have to buffer the entire file from the submitter before sending it to the clients.

Not at all. Use stream.pipe(). I've done it countless times and it works really well.

Not really P2P - I see that you're aware of this and working on it. Really cool stuff.

Why can't I drag-> drop files in?

Both of those things are TODO.

Drag and Drop was working, but I broke it again (file input is handled differently in different browsers, which was a headache) and I just wanted to get some more eyes on it.

As for True P2P - give it a few months. Once WebRTC is here, this will be very very easy to turn into proper P2P.

So is this not "true" P2P because a node.js server sits in the middle?

exactly

Having a central server sit in the middle that all traffic goes through goes right against the very definition of P2P.

Great stuff, and maybe also a step towards a content-centric web, which is indifferent about where content comes from, only that it's precisely what the audience requested.

Though WebRTC might be the 'right', standards-based way for peer-to-peer web communications, I'd be wary of depending on it arriving, with the right features, in any necessary time frame. Incumbent-organization standards efforts often don't deliver with the boldness and promptness that edge innovation needs.

Maybe you'll need to give (some subset of) users a plug-in or node.js daemon, that runs alongside their browser as a connectivity hack-around, in the meantime. Aesthetically impure, yes, but such compromises are another kind of 'schlepping' that sometimes enables breakaway project success.

I'm not sure I follow. This new service requires cooperation with a specific browser on the other end of a centralized service in order to get any content at all.

Compared to a uniform resource locator on top of a dynamic, distributed naming system connected by a robust, global packet-switching network, I'd say this new system is way less "content-centric" and way more sensitive to the content's source.

You're right; I'm projecting my own hopes about what a pure-JS, all-in-browser peer-to-peer sharing stack could do, rather than considering just the specifics of this quick-and-dirty shared folder approach (which isn't even yet p2p, though that's clearly the hope).

Once the remaining browser pieces are in place (via either WebRTC or another shim), a content-centric DHT beyond just shared folders could be the next step. You could have "trackerless torrenting in a tab". Or even "Tor-in-a-tab".

Kudos on the site!

In an earlier version of Sendoid, we were doing exactly this to handle the "larger than reasonable" in-browser sharing issue. We had some privacy concerns at the time with still requiring what was essentially a data proxy server in the middle and eventually removed the functionality when the desktop application was released.

WebRTC is really exciting from a private data transfer perspective and will hopefully start getting us out of the "server-in-the-middle" architecture that we've had to live on for so long with in-browser data transfer. As an added plus, it would be another nail in flash's coffin for live video chat, which is really just a live data transfer problem after all.

Do you know of a way to save the files without prompting? DOM localstorage is tiny. If there was a way to pull updates from your friends automatically and cache them, that would make the web browser a real powerful tool.

There is a 'FileStorage' API which I would like to use rather than the DOM, but only Chrome supports it for now, and I would like this experiment to remain cross-platform (there are enough 'chrome-only' experiments on the web already).

I have opened an issue about this here: https://github.com/Miserlou/DirtyShare/issues/3

Do you have a link to this API? Googling for "Chrome FileStorage" doesn't produce any relevant results.

I think I meant FileSystem instead of FileStorage.

This is the page I used a lot for researching this project: http://www.html5rocks.com/en/tutorials/file/filesystem/

I really like the concept. I often have files i want to share with peers, and they're often in weird places on my desktop, not to mention are often quite large, and i often dont want to have to manage sharing, or move them into my public folder on my Dropbox so a solution like this would be pretty good. That said, i quite like the server in the middle model, since it could probably be tweaked to host the chunks of data if my PC goes offline?

Try http://ge.tt

This is a good example of what the OP is trying to replace. Current file-sharing solutions almost exclusively follow the same pattern: You upload a file which some server stores on disk, then a friend gets a url and can go download it.

Presumably this new service is less expensive to run because there's no disk storage involved, just a central server providing bandwidth and a little bit of RAM to buffer with. And presumably with WebRTC it won't even need that, and will just become a service to easily connect browsers.

Great work. Do you guys host this yourself on EC2? Also, would be great to get SSL transfers.

Hmm, if this will be a real p2p, then your data would not transfer thru server, so developers must to implement SSL(or some other kind of secure connection) in their client-side software.

Reply on news.ycombinator.com