Comment by IsaacSchlueter | Hacker News Reader

Comment by IsaacSchlueter | Hacker News Reader

Parent Full thread

I've heard this before. "Oh, callbacks are fine for little things like this hello world demo, but for Big Programs, you need (threads/coroutines/etc.) because they're Serious Business."

That might be true. Maybe for Big Programs, something else is better. But I think the problem is that setting out to build a Big Program for Serious Business is Doing It Wrong.

The best frameworks (or, at least, my favorite frameworks) are collections of small tools with consistent interfaces and interchangeable parts.

If you can manage callbacks for a 100-200 LOC program, then you can build a Big Program out of such modules with a little bit of forethought. Instead of setting out to write a Big Program, why not try to figure out how to express your Big Problem in terms of a bunch of Little Problems, and then come up with a way to have Little Programs assemble.

Then, assemble the Little Programs to solve the Little Problems. There is no solveable Big Problem which cannot be broken down into some finite set of Little Problems.

Callbacks make it natural to solve problems in this manner.

I actively develop a several thousand line NodeJS project. It's quite nice, actually.

I don't know how to completely address your post b/c it responds to assertions I didn't make. I said nothing about "Serious Business"--and bundling threads and coroutines together is like bundling horseshoes and bicycles. I made no defense of threads.

Callback style, within an I/O handler, is (almost always) a concession to the event loop. When I'm writing a program, I write:

  routine():
    result = do_one_thing()
    do_next_thing(result)
    do_whatever()

With callback based I/O, you need the routine to end so the "reactor" up the call stack can get control again to do I/O on your behalf:

  routine_1():
    return make_some_io_request(callback=routine_2)

  routine_2():
    return make_another_io_request(callback=routine_3)

  routine_3():
    return send_response_to_original_socket()

Now, if you were desperate, you could just call over to the reactor like this:

  routine():
    resp = do_io_for_me()
    next_resp = do_more_io_for_me(resp)
    return transform(next_resp)

.. but, there are (at least) two problems with this:

1. You're making the stack deeper every time you "call over" to the reactor... you will stack overflow eventually

2. In most languages, the reactor doesn't have a way to "call back into" your stack frame and resume it.

Coroutines are a way to "call over" to the reactor, have your stack state frozen (in heap space, without going deeper in the call stack), and then "unfreeze it" later on, when the reactor has completed the IO.

The greenlet package provides this for Python; Haskell and Erlang have native support for this. And, this is ultimately how you want to program. A(); B(); C(). Sure, there are exceptions for UI driven callbacks, and callback patterns aren't all bad, but typically in network IO cases, the callbacks are explicitly a concession to the I/O loop's need to be scheduled, made necessary by a missing language or vm feature.

And: cleaner code benefits 200 LOC apps as much as it does 200k LOC apps. A good design is a good design at any scale.

Why not something like this?

    function routine () {
      doIOForMe(function (resp) {
        doMoreIO(resp, function (nextResp) {
          transform(nextResp)
        })
      })
    }
    
    // or with a little helper function:
    
    function routine () {
      chainTogether
        ( doIOForMe
        , doMoreIO
        , transform
        )
    }

The "Step" library from Tim Caswell (creationix) lets you do a LOT of very interesting parallel/serial stuff. In npm, I have a few helper functions that make it easy to do serial "this then that" stuff.

It would be nice to have something in the language that does this. Maybe something for the Coffeescript guys to check out. For me, plain old JavaScript is enough, I guess.

Could you be a bit clearer about what's going on in your third example?

I think he's saying that the main event loop has called a callback, only to have that callback either call the main event loop, or something similar that processes events until a certain condition, and that in turn may cause other callbacks to be called. Stack depth has increased. And it tends to lead to bugs because sometimes unconsidered things happen in the midst of the first callback's processing.

I plan to prove this point with code. Please stand by.

Reply on news.ycombinator.com