Comment by dwheeler | Hacker News Reader

Comment by dwheeler | Hacker News Reader

Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems.

"small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code."

Source: "Vulnerabilities in the Core: Preliminary Report and Census II of Open Source Software", The Linux Foundation & The Laboratory for Innovation Science at Harvard, by Frank Nagle, Jessica Wilkerson, James Dana, Jennifer L. Hoffman. https://www.coreinfrastructure.org/programs/census-program-i...

(This is a contrast of citations from other works; you can see the specific citations in the report, but I'm trying to do this from a phone. In any case, the issue is the massive difference between JavaScript and everything else, and I think this quotation shows it nicely.)

iOS dependencies in Cocoapods are including less than 1 other packages on average (1). The most I see are Alamofire (a networking package people keep using for reasons that I do not fully understand) and Starscream (a Websocket implementation, that's pretty hard to write from scratch to be honest). Having unnecessary dependencies is pretty much frowned upon.

One of the reasons is that the iOS SDK is really complete by itself while Javascript doesn't even ship with a usable Date implementation. If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically.

(1) totally based on gut feeling, not based on hard data

Java didn't have a decent date implementation until Java 8.

> If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically.

As this article points out, I think that lodash is that for many people.

From what I read about it it’s more like syntactic sugar, not something alike the .Net core libraries.

Get your JS Core libraries once from a repo, reuse them in every application that uses the same ones

Java has had exact datetime implementations since what, 1.3? 1.2? Sometime in the ‘90s anyway. They weren’t fun to use but they worked fine, had the concept of multiple calendars etc...

JS’s version of Date in comparison has always been a toy.

JS's version of Date is pretty much exactly the same as java.util.Date: just a thin veneer around a long representing ms since the epoch. Yes, Java has had Calendar and DateFormat for years, but the API is generally recognized as being pretty horrible (how many bugs were caused by Date and SimpleDateFormat not being thread-safe?)

So the Java community coalesced around Joda time, not unlike how the JS community coalesced around moment.js. The big difference is that yes, Java eventually rolled these lessons learned into the java.time API, but now you're effectively stuck with 2 standards, Joda and java.time. Not sure if that's much of an improvement.

Google Apps Script! Heh...

What makes the js Date implementation unusable in your view? IMO it works pretty okay for most needs.

Why are small packages bad? Independent functions should be versioned and distributed independently. Otherwise we get several utility packages which are nothing but collections of independent functions.

Widely used packages should form the basis of a standard library that is distributed with the language itself.

> Why are small packages bad?

Small, separately maintained packages are bad in a an ecosystem where a final system can incorporate only one version of any given package because it increases the number of opportunities for version conflicts.

Now, it's true that there are some other concerns which weigh in favor of packages being at the minimum useful size, but those necessarily also weigh in favor of a package management system which allows each package to isolate its upstream dependencies without constraining it's downstream users. And not just allows, but facilitates it so that it is the norm, so that dependency conflicts aren't a thing.

I was under the impression that in the npm ecosystem, packages are reused if the version dependencies are compatible, and split if they are not.

Every separate package adds overhead and another maintainer that you've got to put your trust in. I'd rather have few packages of well trusted maintainers instead of a thousand packages from god knows who.

Personally, I've made it a habit to not add anything that requires trivial one liner garbage packages, which amounts to not installing any dependency that uses anything from Jon Schlinkert (so no Webpack). Unfortunately I'm stuck with gulp right now but with the next major rewrite, I'll also get rid of gulp and its 300 dependencies.

> I'd rather have few packages of well trusted maintainers instead of a thousand packages from god knows who.

Then the problem is the fact anybody can submit a package, not small packages.

Linux distributions solve this problem by having dedicated maintainers. Users of a distribution trust its maintainers when they use it. Software developers want the complete opposite: language-specific packages, instant and unrestricted package publication, containers, etc. Nobody wants to have to talk to a maintainer in order to get their software included in a distribution. Of course the result ends up being a mess.

What if Node's maintainers decided to distribute a curated set of packages alongside Node itself? The size of each individual package wouldn't really matter.

> What if Node's maintainers decided to distribute a curated set of packages alongside Node itself?

First, front-end has nothing to do with Node. Secondly, let’s say there’s some sort of front-end foundation who decides to provide a curated set of packages. A natural candidate would be create-react-app, which pulls over a thousand dependencies. That’s just one package you want to include. How the hell do you even start to curate such a beast?

> First, front-end has nothing to do with Node.

It was just an example.

> A natural candidate would be create-react-app, which pulls over a thousand dependencies. That’s just one package you want to include. How the hell do you even start to curate such a beast?

I don't know. How did Linux distributions do it? Arch Linux has over 10 thousand packages. Debian has over 50 thousand packages. Maybe people should ask the maintainers.

Point to me one package in Debian that depends on thousands more.

Oh and I was a maintainer for a major package manager back in the day for crying out loud (was responsible for overseeing the general health of the project as well as directly responsible for maybe a couple dozen individual packages). Never seen this kind of madness.

I don't know about Debian but Arch Linux has huge package groups and metapackages.

https://www.archlinux.org/groups/x86_64/kde-applications/

https://www.archlinux.org/groups/x86_64/pro-audio/

The Arch Wiki recommends the use of these huge packages. I don't see any reason why it wouldn't scale to a thousand packages or more.

I'm talking about one actual package that largely runs in a single process (or at least a single process group) with thousands of deps, not some sort of loosely related by category or by using the same framework (hell, we're talking about the framework itself here) package group where failure of one no name package doesn't affect anything else.

Groups are very different from dependencies. And a big meta-package is there for user convenience to get a bunch of stuff. It's almost never going to be depended on; if something needs a package or two from the group it will depend on that directly.

Have you considered using Rollup? I've personally found it delightful.

Already using rollup as a replacement for Webpack, but it seems to lag behind from time to time. Couldn't use certain new javascript features right away since it would throw an error.

Small packages means more packages, and more packages compounds the dependency graph which:

- makes dependency resolution harder

- means more points of failure

- makes auditing difficult

- makes install time longer

- leads to harder maintenance and upgrade

This also causes heterogeneity in the mass of your dependencies:

- it splits the resources for documentation, testing, tutorial, etc

- it ensures very weak integration between various building blocks, forcing everyone to rebuild glue code again and again

- it makes discovering the proper solution to your problem harder as you must chose a combination of dependencies instead of one

- it makes contributing to the project harder, especially to junior profiles, and increases the price of on-boarding

- eventually, it leads to a culture that pushes the "libs versus frameworks" so far you never get a decent framework for anything. This is why there is no JS equivalent to RoR or Django.

There is, of course, a balance to reach. You don't want a lib that does everything.

The package manager also has evolved to handle lots of dependencies.

In the article 19000 packages are installed in 40 seconds... My Django project with 100 times fewer dependencies takes longer in CI

Npm comes with an audit tool

I have never seen dependency resolution fail (since packages can have private copies), unlike pypi or rubygems.

So some of the downsides to a large dependency tree are mitigated. I'll add one more downside:

- more chances for shenanigans like left-pad to cause issues.

Don't get me wrong, I think 19000 dependencies is fucking nuts. NINETEEN THOUSAND.

> 19000 dependencies

Doesn't this happen because of package manager duplication? I think npm lets packages have their own copies of their dependencies. Since we have a lots of small, widely used packages, they get duplicated at numerous points in the dependency graph. The number of files and installation size explodes.

A big part is the development toolchain, which would be installed at a system level for most languages.

I started a vue project last night, npm install --production installs 4 packages. With dev dependencies, I get 2300 packages. Eslint, babel, webpack, etc bring in lots of luggage

BTW, I think 19000 is wrong, on a fresh node_modules I get:

    $ npm install gatsby
    ...
    + gatsby@2.20.18
    added 1773 packages from 733 contributors 
     and audited 23706 packages in 52.317s
    $ du -sh node_modules
    245M node_modules

Not sure where the "audited" number comes from, but its not the number of install packages. I get 2737 directories containing a package.json, 1477 of which are uniq.

`debug` appears 32 times!

This is what happens with a wafer-thin standard library.

Yes and no.

Because of the small stdlib, you need dependancies, sure.

But why this idea that each dependency must be very small, forcing you to install a hundreds childs, which, themselves, will install a hundred until the pyramid of files start to turn node_modules into a stress test?

The problem is not small packages or huge dependency trees. It's the fact widely used packages have to be installed at all. This is compounded by the package manager which allows packages to have independent copies of their dependencies.

We should have a de facto standard library that gets distributed along with the language's reference implementation. Widely-used packages that have a stable and well-designed interfaces should be distributed by default. If something isn't being used anymore, it can be dropped from the standard library while still being available from the repositories.

The problem is not small packages. The problem is there are many of these small libraries trying to solve the same problem (extending the small standard library more). And when there are no clear winners your dependency tree will be full of packages trying to do the same.

One idea that probably got encoded in the culture is that this code would also have to ship to the client. By having small deps, nobody had to wait around for more advanced tree-shaking (and its idiosyncrasies) to enter the picture.

Make sense.

Which is weird. I would have expected have the leftpad scandal that the community would have found a balance.

To me, culture is a feature, and despite using JS a lot since it has a monopoly on the browser, the community culture never clicked with me.

I often feel like they are reinventing the wheel, not learning from other techs solving the same problems, under and over-engineering, regularly guilty of the XKCD 927 syndrom.

Few projects taste integrated, working together, clean. The stacks looked like hacked together. Agreement on best practices and conventions are timidly showing up. It also seems popular tools are just starting to stabilize and the breaking modifications rhythm to slow down. At least more than anywhere else I worked.

There is no way I can be objective saying this, but anyone with the same feeling?

There are several explanations:

1) JavaScript is a first language for many, many new developers, especially the ones not coming from a formal computer science program. This is because it's multi-paradigm, extremely hireable, runs on everything, runs on the web specifically, and other reasons. If you're only going to learn one language, it's a very good and versatile choice. So its community has a lot of inexperienced members.

2) Because it's such a focal point of the industry, it gets a lot of attention from major players as well and has been evolving rapidly in terms of core features. These features usually get tested out first in the form of packages.

3) Being a dynamic language it's very easy to extend and hack. The prototype system even allows you to modify the behavior of its core primitives.

4) The more experienced devs who use it are often dissatisfied with the qualities of the base language, and so take advantage of its flexibility to try and extend it into what they think a good language is (see TypeScript, Ramda, RxJS, etc).

The response to the leftpad scandal was for NPM to disallow packages being unpublished once they've been out for 72 hours, which really should've always been the case.

Reply on news.ycombinator.com