substack

I suspect that previous attempts haven't been successful for a number of reasons but especially because they tend to be synchronous. Node is a great platform for asynchronous rpc because the asynchronous character is already designed to make the programmer have to think about where the high-latency barriers are. I ran into this problem with drb in ruby for a job queuing system and had to write my own thread pool and polling system to compensate for it. Handing a callback off to your remote method does seem to work reasonably well at addressing these common rpc pitfalls.

Nowjs seems to have a replication system baked in with harmony proxies so when you update data structures these changes get pushed out to the clients. I chose to have dnode just make copies but I do plan on writing a replication middleware at some point with the .use() functionality that landed in dnode v0.5 so I'm interested to see how well this replication approach turns out with nowjs.

And in the next release of Socket.IO Guillermo has promised us multiple channels, so projects like nowjs should get channels for free!

On that note, (browserify author here) I would love to see projects like nowjs start to abuse the "browserify" field in the package.json for client distribution like dnode v0.6 does (presently in staging until socket.io-node v0.7 lands). With this field you can `require('pkgname')` in your browser-side code just by specifying 'pkgname' in your require list, even though pkgname might have both server and browser-side components.

The solution I settled on for dnode is that a deep traversal finds all the functions in your argument list, pulls them out, and wraps them. This part of the protocol is a reusable module if you want to go with it:

https://github.com/substack/dnode-protocol

It's written for continuation-passing style but it shouldn't be too hard to retrofit it for other call styles if you want to preserve explicit returns for instance.

Rival node.js RPC project (dnode) author chiming in here.

This is pretty neat and I like the emphasis on getting stuff going in as few lines as possible but I'm not convinced that return values and implicit result callbacks are the way to go for asynchronous requests, which is how I understand this project to work. Often when you want to make a remote call you'll be doing some I/O on the server side, which in node is asynchronous so you can't return right away. Passing along a callback from the client side works much better when the I/O actions have some values to call the client back with.

Ah, your point makes much more sense now. Yes, the author had some overly complex examples, but I'm optimistic about using user-space libraries to handle flow control without having to add new language primitives.

First, why wrap your second example in try {} catch {} but not the first? The second example (final example from the article) seems like it was intentionally made more complicated to prove its point, but it could just be unfamiliarity. Node idioms are just different from other environments and it takes time to get a good feel for them.

(Edit: noticed that the article does it this way.)

How about just:

    function insertCollection(collection, cb) {
        if (collection.length === 0) cb(null);
        else db.insert(collection[0], function (err) {
            if (err) cb(err)
            else insertCollection(collection.slice(1), cb)
        }
    }

That's hardly much more complex, and as a bonus won't lock up your whole program while db.insert() is working. You could use something like my own library Seq() too:

    var Seq = require('seq');
    function insertCollection (collection, cb) {
        Seq.ap(collection).seqEach(function () {
            db.insert(c, this);
        }).seq(cb).catch(cb)
    }

That is, if you actually want the inserts to go sequentially. Usually you want them to go in parallel with say a maximum of 10 pending requests:

    var Seq = require('seq');
    function insertCollection (collection, cb) {
        Seq.ap(collection).parEach(10, function () {
            db.insert(c, this);
        }).seq(cb).catch(cb)
    }

I don't see any "fundamental issue" with node here. It's just a very new ecosystem and the idioms and libraries are rapidly evolving.

There's also the pushd and popd commands which are part of bash already.

And there's a wikipedia entry for this even: http://en.wikipedia.org/wiki/Pushd_and_popd

Referencing objects isn't a problem since dnode doesn't do replication, it just makes copies, but referencing functions when the client disconnects can occasionally be bothersome. You do get a hook when the client disconnects:

    conn.on('end', fn)

so that helps. At browserling.com all of our backend and frontend spanning lots of processes over several servers is tied together with dnode, and we haven't had any big problems tracking references that I can think of.

You can pass functions around in your JSON.

I can think of a few, but "linear/nonlinear" doesn't seem like the right way to look at it. From what I've read, productivity has more to do with statistical distributions: http://xw2k.nist.gov/dads/HTML/lotkaslaw.html

Very curiously, these distributions are self-similar in that even when you restrict the sample to top-tier academics for instance, they still hold.

Also check out node-binary: https://github.com/substack/node-binary which has a similar interface but nicer internals and only needs to keep 2 buffers in memory at any given time instead of a linked list like bufferlist.

I wonder if the decrease in brain size is due in part to infant mortality and trauma in childbirth. Big brains don't exactly... fit as well coming out, although it's also not clear how much early brain size influences brain size later in life.

For extra fun, type this in the address bar while on the github 404 page:

    javascript:var theta = 0; setInterval(function () { theta += 0.05; parallax({ pageX : (Math.sin(theta) + 1) * 1000, pageY : (Math.cos(theta) + 1) * 1000}) }, 50)

If America is in a period of Byzantinian decline then how do the great tech-hub city-states like San Francisco and New York play into this narrative? This doesn't seem consistent.

Frankly I've found python to be quite antagonistic to functional programming with very weak lambdas and poor building blocks for building interfaces in monadic pipeline style. I guess coming from java it would seem refreshing but coming back from haskell or even just javascript it feels like a straitjacket.

At StackVM we use an IRC bot to tell us when the servers are down. In our case the bot is just a DNode service so all the backend needs to do is connect to the IRC service, post its message, and disconnect.

http://github.com/substack/rowbit

It's a pretty great fit since we already use IRC a ton to coordinate development so we see the notifications right away.

I like how there's a command line interface plus an application interface, so I can tie forever process management into my web interface and backend control interface. Right now the processes are just sitting in a `while true` bash loop.

Which browser are you using?

Glad you like it! Easter egg: http://browserling.com/queue

Sorry about that! When the API servers go down (these interface with our windows servers) it tells everybody in the queue "time's up" instead of a more informative message since we're using connection drops as a poor way of passing along the session expiry event.

Since the article uses evolution metaphors, how about this: forks can be a good way to help organisms (read: projects) navigate a complex multivariate fitness landscape. Abandoned forks then serve as concrete markers of particular design decisions that can be revisited later if a similar kind of problem comes up.

We often don't know ahead of time how a decision will work out, so calling these forks "wasted effort" seems to miss their utility as experiments that can drive the direction of future development indirectly.

Instead of reading in static files yourself, you can just use connect (or s/connect/express/g and it'll also work):

    var connect = require('connect');
    var server = connect.createServer(connect.staticProvider(__dirname + '/static')).listen(4000);

Pow, that was easy! Plus, since connect (and express) are just up the prototype chain from http.Server, it works great with socket.io.

Each approach has liabilities that should be recognized. Intuitive decisions might "feel right" but there are so many cognitive biases that can make something intuitively appealing but surprisingly wrong. Analytical, rational approaches are likewise poor at developing creative solutions but is good at removing bias from reasoning by clearly stating every step.

RPC is a domain where functional patterns really shine since the high-latency nature of network traffic is well suited to asynchronous, callback-oriented event systems which are themselves inherently functional.

My own RPC system enforces continuation passing style exclusively to unify the calling and return-value semantics. With this functional pattern in place it was pretty straight forward to transparently wrap arbitrarily many closures arbitrarily deeply in callback argument lists and method requests (which are the same thing under continuation passing style).

I personally can't stand office chairs. Give me a sofa and a short table or a barstool and a counter, ideally both so I can switch between them, and I'm content. The article just assumes that everybody works best in chairs, but there are variations in how programmers best get work done. It's not simply a matter of running a cost-benefit analysis on how much to spend.

I've messed with TreeBuilder before but it looks like I should give this XPath business a look the next time I need to parse some HTML.

I was a bit confused when I hit the "quote-likes" section until I realized it was an aside however. A different colored box might be good for those.

This article says nothing about how lifespans are far longer now, as the comments section points out. Cancer is what you die of when everything else fails to kill you. Cancer is only "purely man-made" in the same sense that increased life expectancy is "man-made" too.

In this case it was a derivative work based upon source code from the book, which is permitted under the terms but precludes open dissemination. Further, the rest of the codebase was very tightly coupled to the algorithms since since the algorithms updated state in-place instead of returning values, so dropping in an open version of these algorithms was very non-trivial.