CommonJS Is The Future For Rails JavaScript Packaging | Hacker News Reader

CommonJS Is The Future For Rails JavaScript Packaging

alexmaccaw.co.uk

75 pointsmaccman18 comments

As the creator of both Sprockets, which underlies Rails 3.1's asset pipeline, and Stitch, which the poster of this article ported beautifully from Node.js to Ruby, I feel obligated to reply here :)

While CommonJS modules are undeniably elegant, they carry with them a high conceptual overhead. It would be unreasonable to expect Rails developers to rewrite all their existing JavaScript in order to take advantage of the asset pipeline. Most people do not want to manually import every dependency into every source file (and in fact Rails goes to great lengths to make use of Ruby's "require" method largely unnecessary).

Sprockets solves this with the "require_tree" directive that automatically pulls in all source files in a given directory. If you've been keeping your JavaScript source files in public/javascripts, it's trivial to upgrade your app to 3.1 and take advantage of the asset pipeline — just move your source files into app/assets/javascripts. Then when you do need more specific control over source file ordering, the "require" and "include" directives are there for you. Combine this with the implicit closure around each CoffeeScript source file and Sprockets gives you 80% of the benefit of CommonJS modules with 20% of the complexity.

Stitch (and CommonJS in general) does not address the problem of CSS preprocessing or concatenation, and does not provide a mechanism for managing other assets like images and sounds. Sprockets supports Sass and SCSS, and provides a global load path which lets you depend on JS, CSS and images from a single logical package — even a Ruby gem — with essentially no configuration.

I hope this clarifies why we chose to go with the Sprockets approach rather than CommonJS for Rails 3.1.

Sprockets can be used for JS, CSS, images, and so on, because it is so simple: it is "just" concatenation. I understand and agree on the benefit/complexity argument.

But the fact that this method works universally, is also the reason it cannot accomodate individual needs for each "asset type" (css, js):

* For Javascript we cannot use CommonJS (or something similar)

* For SCSS we cannot use variables, mixins

For me, this means a lot. Traditionally, Rails has been giving me the best web environment experience I could imagine. With Rails 3.1, I can't help thinking: Why did they leave some of the best parts out?

I know I actually CAN use SCSS variables and mixins - I just have to use SCSS' @import rather than Sprockets own =require. But that leads to my stance in this matter: If we need to bypass Sprockets in order to get the cool features, then why use it in the first place?

We should use a system designed specifically for JS to management our JS, and a system specifically for CSS to manage CSS. Stitch and SASS comes to mind of course.

While I understand the benefits of the "Sprockets way", I'd like to see Rails use specific tools for specific asset types in order to give developers easy access to all of the available features instead of just 80% of them.

Didn't Ryan Dahl say he wish he used something else for packaging Node modules?

(I don't know what he had in mind other than CommonJS and I don't remember where I heard it so this is pretty hazy...)

Yes, it's in this interview:

http://bostinnovation.com/2011/01/31/node-js-interview-4-que...

BostInno: Is there anything you wish you had done differently with node?

Dahl: Yes – many things. For example, I wish I had not used the CommonJS module system. It’s far too complex and wildly different from how the browser works.

In another interview (which I can't find), I recall him worrying a bit that CommonJS is not async, though admitting that a few lapses in async purity might be fine. In any case, I'm not aware that Node actually has any plans to move away from CommonJS. At this point, a whole hell of a lot of code relies on that style of require. Perhaps it could be swapped out in some backward compatible way, but it seems like a lot of work. (Also from the point of view of someone who just uses CommonJS style rather than implements it in something like Node, I have to say, it's dead easy and intuitive. Why fix what isn't broken?)

Wow, wtf? I got voted down for finding someone a reference? I'm not making an anti- (or pro-) CommonJS argument here. I just found the reference.

This may go some ways to solving the issue (mostly with asynchronous module loading) - https://github.com/joyent/node/commit/9967c369c9272335bb0343...

Like you say, I don't see any issues with the current system (and don't like wrapping everything manually in define function). You'll always need to get a server involved at some point for concatenation and performance reasons - and I think the current system works fine.

Ah that makes sense. (As I say, I remember that he said elsewhere he wasn't thrilled about requires being synchronous.)

There's this huge misconception that because "require" synchronously returns the module object it must use synchronous IO to load the modules as well, which is not true. The root module and it's transitive dependencies can be loaded asynchronously before any execution of the code (and thus require calls) takes place.

To do that the dependency module IDs must be extracted from the module text via static analysis, but that's fairly easy since module IDs are specified to be string literals. A simple regex is usually good enough, but using one of the many JavaScript parsers is safer. For cases where the module ID needs to be computed or loaded based on a runtime decision, an asynchronous form of require should be used.

The expectation is ECMAScripts modules will map pretty cleanly to CommonJS modules, but with real syntax rather than function calls. In the meantime CommonJS modules are a pretty good compromise I think. Simple implementations can do the loading using synchronous IO as "require" is called, while more advanced loaders (especially for browsers) can implement a slightly more complicated system to do the loading up front before excution.

> To do that the dependency module IDs must be extracted from the module text via static analysis, but that's fairly easy since module IDs are specified to be string literals. A simple regex is usually good enough, but using one of the many JavaScript parsers is safer.

I did exactly this for browserify, yet another node-based browser-side require() bundling system. Another benefit of this approach is that you can use modules made for node just by require()ing them in your browser-side javascript, so long as your static analysis bundling is recursive and the modules can be `require.resolve()`'d by node.

If only it were true. I started in on Sprockets to get my feet wet for Rails 3.1, and have to say I was disappointed. Sprockets is a textual concatenator, which makes it appropriate when you're writing short snippets throughout a project. By contrast, CommonJS (and requireJS and a slew of other implementations of asynchronous modules) implements well scoped modules, which have all kinds of boons.

"If you've ever use Node or Python you've used CommonJS modules"

Huh? Python, really?

I could be wrong, but I imagine he's referring to the module concept. Not CommonJS itself. (Or perhaps referring to skulpt.org, but I doubt that.)

Yes, that's correct. Importing is similar to require(), although there's no explicit exporting.

setting module-level __all__ can modify what gets pulled in a wild import, but that's about it.

It's also used by `help()` (and pydoc): if there's an __all__, help will only pull the docstrings of the stuff in __all__. `dir()`, on the other hand, does not care.

CommonJS modules are modelled on python's I think.

With CoffeeScript's destructuring assignment, you can even simulate Python's "from" clause:

  foo = require 'foo'
  {Bar} = require 'bar'
  {X, f} = require 'abc'

Compiles to:

  var Bar, X, f, foo, _ref;
  foo = require('foo');
  Bar = require('bar').Bar;
  _ref = require('abc'), X = _ref.X, f = _ref.f;

Looks very cool, will play around with it. Seems like a big missed opportunity that Sprockets doesn't speak CommonJS.

Sprockets, as it exists, can't do module loading: it's a Javascript concatenator. All it does it copy the contents of <filename.js> in place of //= require 'filename.js'.

Gotta, say, I just spent a chunk of time converting a project from one to the other (actually RequireJS), and I really think it was the right move.

Reply on news.ycombinator.com