Comment by kevingadd | Hacker News Reader

Comment by kevingadd | Hacker News Reader

The response by Isaac filled me with rage as soon as I got to 'No one actually writes code like this'. The guy provides a real example from his application, describes how alternative solutions stack up, and how he'd like to solve the problem, and gets called out for presenting a 'strawman'. Sigh, newsgroups...

I personally share the original author's perspective on this and was disappointed to see more sophisticated methods of writing concurrent code (futures, coroutines, etc) dropped from Node in favor of raw callbacks. In my experience, chaining raw callbacks is too easy to get wrong for it to be considered a reliable way of writing all your asynchronous logic, even if it is a great primitive to build a framework on top of. In particular, the effort involved in propagating exceptions all the way through a callback chain is rather significant, and making a mistake in a single function undermines the reliability of the entire chain (usually resulting in work silently being dropped on the floor). I wish more of an effort was made to address this problem.

You have seen what the author posted later on in the discussion?

https://github.com/sam-mccall/node-plate

There are soo many interesting ways to solve this: http://news.ycombinator.com/item?id=2101724

Is that the wrong link, or are you threadjacking?

Wow, wished I saw it earlier (can't edit now). Bad paste. Correct link:

http://www.infoq.com/articles/surviving-asynchronous-program...

Link unrelated

I like this syntax more: https://github.com/creationix/do

I like neither.

They're still riddled with boilerplate (anonymous functions) and are quite horrible to read and write (bracket nesting).

I sometimes wish node.js (and js in general) would adopt co-routines. They'd seem to be the perfect middle-ground here; async code that looks almost like threaded-code, and no more callback-hell.

And later still:

https://github.com/sam-mccall/node-ship

(supports a few more things than just method chaining)

I wholeheartedly agree with you that the comment "No one actually writes code like this" was inaccurate and not useful for the discussion.

I see people doing this exact same style of code in EventMachine in ruby, and have done for years. I am continually having to advise newcomers to the evented space that they need to write state machines inside objects to maintain state and process flow, and advising them to wrap callbacks into message sends to these process containers.

There is nothing lacking in javascript to do this (create state and function containers), just as there is no lack of OO in ruby. Despite this, newcomers to evented programming always do the same thing. They revert back to a primitive style of programming, writing very long, very tab-deep functions. This is bad practice, you don't do it in an imperative synchronous environment (after some experience), and you shouldn't do it in an async environment.

You do not need to use closures for state passing. You shouldn't use closures as general state containers. They're even worse than using stack frames, as they're even less easy to debug (at least walking stack frames is easy with a decent debugger). They're really useful for the minor things, but to encode a complex process, they're completely inappropriate for reasons already discussed (error handling is an ever so common one in the network space).

I think this may be what the person was trying to get at with their comment, but they completely lacked substance, which was no use to the OP.

Everyone seems to end up discussing some new async-useful algorithms or datastructure that will "save the day", but in reality, until you start encoding application logic processes into decoupled, testable containers, you're always going to fight some complexity. This really has nothing to do with async, the principles of encapsulation and coupling are absolutely well discussed and documented, and merely apply even more vehemently to the async space. The unfortunate fact is that many of the users of these libraries do not factor code in this manner, and as such this discussion comes up again and again.

I honestly don't know of any good open source examples of complex async applications in either language (ruby or javascript) that are well structured. I can point you time and time and time again at libraries that are poorly constructed based on the "trivia" examples that "don't require" any structure and so just use a single object/scope and some closures to achieve a minimal use case.

There are many possible reasons for this lack of good examples. The reason I haven't written one is because it still takes some care, mostly in architecture, and no one will pay me to write such open source. There probably are some good examples in C, based on libev, libevent, glibs loop, etc, etc. I'm not intimately familiar enough with any example code bases to be able to call them out off the top of my head.

write state machines inside objects to maintain state and process flow, and advising them to wrap callbacks into message sends to these process containers

Any chance you could give an example of this refactoring / change of perspective (or link to a blog post, etc)? I can almost imagine what you mean, but it sounds like useful enough advice that I'd really like to grok it, and a concrete example would help a lot.

I'm currently working on another approach to reduce the complexity and error-proneness of evented programming. I'm working on a combinator library to abstract and automate the more common patterns of wiring up callbacks and propagating exceptions. Have you seen any success (or pitfalls) from this kind of approach?

I honestly don't know of any good open source examples of complex async applications in either language (ruby or javascript) that are well structured.

So, at first you claim there is clean way to do it, but then you admit nobody is doing it the clean way?

That sounds a bit strange, doesn't it?

I think the reality is that there simply is no pattern to escape callback-hell in complex evented applications. Or why else would everybody still be writing the same spaghetti-code with the logic scattered over at least two callbacks (success/errBack) per unit of work?

I'd go as far and claim the spaghetti-pattern is inherent to evented programming, unless you wrap it all up in something sane like co-routines.

I think what raggi said is that there are no good _open source_ _complex_ async _apps_, mostly because when people are writing open source they tend to write libs, not apps, or end up not working anymore at the problem before it gets really complex.

I've worked on a commercial, complex, async codebase with raggi and have to agree that there is a way other than callback-hell, even for complex evented applications.

In my personal opinion, even more so on ruby than javascript, but I don't want to get to that really.

edit: changed asterisks to underscores because I didn't know they'd be italicized

Still sounds dubious to me. Then why has this pattern not emerged as best practice long ago?

And, frankly, I'd be curious to see a snippet of code in that style.

yea I don't doubt it.

the reason it hasn't surfaced as a best practice yet is probably because best practices for async programming haven't been a hot topic yet either.

and the point is exactly that a snippet doesn't fit the bill. its easy to abstract complexity on a snippet. its only with the full fledged solution that you start to see best practices emerging and making a difference.

Since when was it not a good practice to encapsulate, abstract, and decouple?

Async is a bitch, no matter what people tell you.

Back in 2002 there was a lot of buzz about 'single-process' web servers like lighthttpd, but when I evaluated the open source products that existed at the time, I found that all of them had problems with reliability and data corruption. Matters are probably better today, but it underlines that it's much harder to make async programs correct than to make programs that use blocking I/O correctly.

I've spent some time building RIAs that are a bit more complex than what you usually see in the web using Javascript, GWT, Silverlight and other environments that all use async comm. The reason you don't see RIAs this complex often is that nobody knows how to write them.

A key architectural principle is that GUI elements need to register their interest in changing data... If you try to hand-code the paths that feed information in the GUI elements, you'll eventually reach the point where all hell breaks loose when you make a little change.

Another principle is to design your communication protocol around the application rather than the other way around. Programming in a single address space, the common practice is to build complex functions by composing simpler functions. That just doesn't work in distributed systems, not if you care about performance or reliability. If you can write one RPC call that does the work of 20 RPC calls, you eliminate a lot of complexity in your app and typically speed it up dramatically.

> Sigh, newsgroups...

What does this have to do with newsgroups? You can get this kind of response on any medium, no need to blame it on newsgroup.

Reply on news.ycombinator.com