Hacker News Reader

Comment by davedx | Hacker News Reader

So I think there is a parallel to this with the whole "everything must be in tiny modules on npm" movement in the JavaScript community. If you do this, you end up with lots of repositories, a rigid separation of concerns but a network between you and your dependencies that will get hit a lot unless you wield some npm link sorcery, and a bunch of extra complexity.

A modular monolith application is what people have been writing since people thought up the notion of modules. Enforce proper discipline when building your app out and you won't need these physical walls between your functional areas.

I'm currently reading SICP, and the notion of using "block structure" in Lisp to compartmentalize and encapsulate functional areas of code is introduced in Chapter 1.

Get the basic stuff right before you start introducing complex systems to split up your software.

View on HN

And there's probably a parallel to "one file per type" with tons of little types. It feels like you're making progress and doing stuff. You've not just done one task, you've deployed changes to 20 services. It also feels like you've made things more abstract than they actually are. It also feels more like Real Serious Programming.

I've used a Twitter API client (works fine) that implements itself across 7 separate libraries. All same project, they all work together, and they aren't used in any other system. Just separated things out for the fun of it. One library for Twitter client "Factories". One for Twitter "Credentials". Another for Twitter client "Security". Zero benefit to the user or to the project. But it certainly makes things seem more important, eh?

It totally makes sense to have separate libraries even for one project if they can be implemented by different people and tested separately. I'd shout for that design any time of the moon.

They can't be deployed separately. There's no point in splitting them up. Testing and development aren't affected.

Deployment is not the only thing that matters. Development is also important. I am actually working on an interpreter for a web scraping tool, where I'm working on the interpreter and my friend on the IDE. The likelihood of them ever being deployed independently is low but the separation is helping us focus.

Seems to me like the only concern with tiny modules on npm are development related. npm could improve the workflow here by specifying a single path to look up (autolink?) additional development dependencies rather than having to link them in each project one by one.

edit: And it seems like someone already thought of that and wrote a module for it ;) https://www.npmjs.com/package/npm-autolink

npm is the biggest nightmare of a package management system that I've ever come across.

What other package management systems are you using for comparison?

I know several: Guix, Nix, dpkg, yum, brew, pip, gem/Bundler, composer, package.el, and probably others that I'm forgetting. Most of those I listed are bad (the exceptions being Nix and Guix that actually do it right), but npm takes the cake. It somehow manages to create situations where applications are using many different versions of the same library, and where the same packages are duplicated in the hilarious mess that is the node_modules directory tree. Trying to package software written for Node in a systems package manager for use on GNU/Linux distributions is basically impossible because npm has created a wasteland where even the simplest software may have over 100 runtime dependencies, and then if you actually want to verify that their test suite passes you need 100 more "development" dependencies. jQuery's package.json will create a node_modules directory with 265 node_modules directories in it. [0] 265 separate pieces of JavaScript software to build jQuery from source! That is insane, but completely mundane in Node land. I had a discussion on their IRC channel about the challenges I've had trying to package software written in Node for GNU/Linux distributions, and was told that it's just the "Node way" and that it is better than everything else. Everyone just uses npm without knowing what's under the hood and moves on with their life, but when you actually try to untangle it all so that software written in JavaScript isn't tightly, irrevocably bound to a single package management system written by people that do not understand how to do package management, you find yourself deep in computing hell, wondering how everything went so damn wrong.

[0] http://dustycloud.org/blog/javascript-packaging-dystopia/

So basically, what you're saying is that its hard to package Node apps using the typical systems package managers in situations where you want to build everything from source.

I can live with that. The worst fate would be having to modify what npm does (replacing the package fetch with building from source). Given that npm is licensed under the artistic license, thats not half as bad as it sounds.

I do agree that there is a lot of unnecessary complexity introduced by today's JS build tools such as grunt and gulp. But npm isn't to blame about that - its only sin is creating and nesting duplicate dependencies, something which is already fixed by npm 3 (completely flat node_modules unless there are unavoidable dependency conflicts). The solution here is to avoid bloated packages such as grunt and gulp that try to install everything plus the world. (coffee-script grunt? seriously?)

One thing npm could do is separate buildDependencies from devDependencies. There are a lot of tools listed there that only pertain to testing, and testing a serious client-side JS library such as jQuery properly is a hard and complex endeavour.

Also, I can't help but notice the irony of complaining about the complexity of web application build tools given the complexity hell that are autotools :)

Let's add another problem to the list: The packages uploaded to npmjs.org are typically not source packages, but "binaries" (in that they aren't source). They often contain minified JS files and no test suite, making them effectively useless for anyone that wants to build from source. Of all the language-specific package hosting sites, PyPI is the best I've encountered as they do proper source releases.

Also, a package manager should not be a build tool. A build system should never require a particular package manager to be present. It's okay to say "you can optionally use this package manager to get all the dependencies needed to build", but it's not cool to say "you need this package manager in order to build."

npm does allow you to specify a repository url in package.json and warns when one isn't present.

I guess we may've gotten a little too happy with the centralised proprietary cloudy source distribution channel with a single point of failure...

Reply on news.ycombinator.com