Hello Yarn 2, Goodbye node_modules
freddixx.medium.com
freddixx.medium.com
Sure, node_modules is a behemoth and there is still a likely better way - but I _love_ having all of my packages on a per-project basis. I used to be a user of Yarn 1 and 2, and their debugging and installation system is insanely better than NPM's, but I quite like having everything in one place for every project.
I remember the absolute hell that was python packaging as a beginner and I don't know why anyone would want to move even an inch in that direction.
What's the problem, really? How many projects are you working on that the size becomes a problem? Or is it the fault of Apple charging way to much for large SSD drives? Maybe it could be faster, sure. A good cache could help with that.
Except it's exactly the same. CD into a folder.
Overall, like the OP said, it's possible but it's not the same.
Edit: Though as another commenter pointed out, the patch workflow alleviates these issues so it really isn't that big of a deal, you're right.
My projects are self contained again, even though they use global package stores, and my IDE can treat the package store dir as a part of the codebase for debugging. I love it.
That being said, overall I found all of that more trouble than it's worth btw, certainly for Go. I certainly had to debug and fix far more issues with the container software itself than with issues derived from diverging environment issues, especially on coworkers machines who aren't very familiar with all of this and are "only" developers (nothing wrong with that btw).
I like that.
Working on multiple projects that use the same dependency and you notice something and want to test it in node_modules like op described, and don't revert it now you're wondering why one project is broken.
I think quite the opposite - in fact it'd be somewhat unexpected to hear very much or often about an established tool that is just boringly doing its job well.
This is the pinch of salt you need to apply to anything you read about a new tool / approach to something in tech. New ideas are sometimes genuinely better than the old way, but far less often than at least someone claims they are - which is always.
Equally, you can't take people not leaping to the defence of the old way as signal either. There's little incentive to do so - if you're using the old way and happy with it, there's no need to convince anyone of its merits. It's already established. Better to just spend your time building stuff.
You can do that.
You can go in the dependencies and open it files, you just need a vscode extension because vscode can't open files in zips by default.
For modifications, it may works, I never tried it.
Also you can also configurate Yarn to use the node_modules linker and you'll have it back.
I do think yarn 2/3 has some way to specially cater to easily editing your npm packages.
This also reminds me how much we keep reinventing the wheel on this. I know it's likely infeasible for various reasons but it seems like it would be great to have a language-agnostic package manager that worked more or less like Bundler and can solve these problems once.
(The most obvious downside of that is that – well – what language is it written in, and does it cause you to have to install multiple toolchains. Maybe the solution is some kind of standard spec for how package management should work, and the manifest formats, etc, and then each language ecosystem implements that standard. Or something. IDK.)
Edit: A cool feature of such a system could be installing dependencies from multiple language ecosystems via one manifest file.
Amen to this. This type of stuff leads to so much confusion especially for beginners. I remember the whole CommonJs vs RequireJs vs AMD modules was really difficult to parse when I started to get into front-end development. Yes having multiple choices lets us experiment with alternatives, but I think we underestimate the costs of complicating the ecosystem as a result.
Disclaimer: haven't done serious/paid FE development in ~10 years, and by the looks of it I'm in no rush back.
Oh man sounds like nothing changed. Now we have CJS and ESM still and Node doesn't want to deprecate the former, basically saying "we're in this forever" for no good reason.
I know one legitimate reason is "tree shaking" (source-level LTO when bundling modules). Dumber, static import/export statements probably simplify that in some way. Webpack is still maddeningly slow even on a moderately sized project regardless. Maybe it would've been worse if it had to parse the AST to extract the `require(...)` calls.
One change that ES modules introduced, I think, for no other reason than to be backwards incompatible, is changing the behavior of the default export (`export default foo` transpiles down to `module.exports.default = foo` instead of `module.exports = foo`).
Other "ohai guys this is the new normal now" kinds of changes are making the dynamic imports async-only (after not supporting them for a while) as well as changing the behavior of module resolution. Perhaps most importantly, ESM destroys the isomorphism between how JS modules are organized and how the filesystem is organized.
And the cherry on top is called TS-ESNode: https://github.com/K-FOSS/TS-ESNode because TypeScript modules and ESM are the same thing yet you need to somehow find this third-party shim which is required for them to work together at all. It's enabled by wrapping the interpreter, just like Yarn2's new dependency resolution.
npm install [package]
yarn add [package]
Even the options for this command are different (--save-dev vs --dev). In fact, the situation is even more confusing, since `yarn install` exists but does not accept parameters:
yarn install
[1/4] Resolving packages...
success Already up-to-date.
yarn install something
error `install` has been replaced with `add` to add new dependencies.
Run "yarn add something" instead.Why? Aren't there multiple package managers in many languages? SBT/Maven, Ant/Ivy/Maven, PIP/Poem(or poetry or whatever it is). What's the harm?
Perhaps I just don't have that much experience with pip, but it feels more like a tool to install packages, then you're on your own with some makefile or python scripts with no standard setup.
In most ecosystems, I can share package sources between a VM and a host.
npm shares mutable content (eg compiled artifacts) with package sources in `node_modules`. That breaks a ton of workflows that are common in other ecosystems.
For instance, in ruby apps using bundler, I can commit my dependencies in vendor/cache; there are no network dependencies other than fetching the source code. That makes turning code into a running server faster and more reliable.
Go supported the same thing from v1 via GOPATH (because that's how google runs their repo). Commit your dependencies and carry on (go had different shortcomings in dependency management in those days).
This feature turns things like `left-pad` from a fiasco into a non-event.
Also, it took npm years to implement a lockfile, only for most npm commands to disregard it. I used to frequently get versions other than the one I wanted. It's been ~6 years since I switched away so it may have improved since then. After my experience using npm I fundamentally do not trust the brand, and until yarn screws up I have no reason to give npm another chance.
In fact, both of these package managers have had direct support for this workflow since 2016 (https://docs.npmjs.com/cli/v7/using-npm/config#offline) and 2017 (https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/) respectively.
Experienced developers they are not.
Bundler, AFAIK, isn't first-party, its a separate team and project.
Historically, IIRC, gems wasn't even first-party
Good standard solutions are often the outcome of multiple competing efforts pushing forward, proving what works, and proving what doesn't.
But, it's irrelevant. I'm not arguing no one should make a third-party package manager, just that there should be a standard, ideally first-party one, that is good and well supported.
Languages like Ruby, Rust, Swift have recognized it is beneficial for the core project to provide a package management solution.
Which is to say, yes, it’s painfully half-baked and riddled with inconsistencies. But that is pretty typical of JavaScript anyhow.
Yeah but what I mean is that as it stands you literally cannot implement a package manager based on ES Modules alone. So it’s strange to me that someone would say that ES Modules are the package management solution for JavaScript. That’s not what it is currently for. Though the relevant standards may certainly be officially extended in the future, to allow for standardized package management based on ES Modules.
In principle, ES modules do away with needing a package manager. And yes, folks are already working on the consequential gaps in capability.
The point still being, however incomplete it currently is, this is the first-party solution. Node is not JavaScript.
At least one time, I tried to get into python packages, but I recall there seemed to be about a half a dozen methods for doing it at the time. I didn't try very hard.
I wish I could say I never had to think about it, though.
pip install pandas
??
The trouble is firstly that pip will happily install conflicting versions of numpy which breaks your code.
This occurs because pip didn't check dependencies until last year, which is one of the reasons it's a terrible package manager.
Now it just gets stuck spinning it's wheels for ages and then errors out, which is better but still not good.
Conda, while really slow actually handles c level dependencies which makes it usable.
So don't!
I consider it a public service to keep mentioning this though, so that in future jobs, I don't have to explain over and over again why I don't want to use pip to "manage" our dependencies.
The problem is not so much technical but rather social in my view. For one, it's quite "natural" for language developers to try to grow their community around its very own tools, even if those tools effectively "cut corners".
The other "social issue" has to do with practices: packagers and users of the generic GNU/Linux tools I mentioned have different expectations in terms of having a curated package set, ensuring common conventions are followed, building software from source, and so on. This is at odds with the practices encouraged by some of the language-specific package managers.
Npm does not try to manage complexity, as explained at https://dustycloud.org/blog/javascript-packaging-dystopia/, Java packages often come with opaque jars that nobody builds from source, sometimes due to circular dependencies, and so forth.
I chuckled a little when I realised that npm, Yarn, Yarn 2, and pnpm can all be found alive and well in the ecosystem
Deno is an interesting model, but maybe not widely applicable?
You might not need to transpile your app (via babel), but you still need to bundle it (via webpack/rollup/parcel/esbuild/swc/etc)
Unfortunately loading my app went from a second to half an hour (I think it just broke loading all the files somewhere).
But, also, you can do quite a bit with carefully selected small dependencies and a minimal amount of JS: I have some relatively complex sites that forgo bundling with no perceptible issues here
We try to write code that is isolated/modular for many non-controversial reasons. A common strategy is to co-locate related functions/classes/logic in one file and keep this separate from unrelated logic in other files. The more your code does, the more logic to build, the more files you'll have. The same holds for your dependencies. The more they do for you, the more files/modules they need to isolate their logic. Thus micro-dependencies have few files in two cases: 1) they do relatively little and shift responsibility to your code 2) they do a lot but have already combined/compiled their files into 1-2 files for you, often using a bundler.
ESM is an excellent native solution to referencing logic from other files so you can split & isolate your code when you write it. But this is an authoring strategy, and it should not dictate your distribution strategy.
I totally agree with you on pruning dependency trees though.
Many project are going to use some kind of central import/export system to keep track of external dependencies and that could perfectly well be the result of a package manager compiling the equivalent of a package.json to a .ts file.
Only-URL imports does not mean that no structure is available
Haven't used Bundler, but have you looked at Nix? It's got support for a bunch of languages/ecosystems, including the npm one: https://nixos.org/manual/nixpkgs/stable/#chap-language-suppo...
Either way, the trope doesn't really apply. Even if this were a new implementation, it wouldn't be a new standard. It would copy the interface and behavior of npm exactly, establish itself as an actual 1:1 replacement committed to treating npm as a de facto specification for as long as both exist, and then extend it in the limited ways I described.
Yarn made npm move in the right direction. Like in any other area, we need competition.
Meanwhile, as much as I respect the work of Mael (yarn maintainer) I’ve switched back to npm. Mostly because:
- isaacs (original creator of npm) did some commits lately
- now backed by GitHub
- they introduced workspaces (monorepo support) in npm 7
- it’s still easier to use the standard tooling in dev/ci/prod
- like many, I had issues trying to convert my projects to yarn v2/3
The only thing yarn did I want in npm is an easy way to enforce the npm version across all environments (https://classic.yarnpkg.com/en/docs/cli/policies/).
I've got actual work to do, I don't have time to spend a day or two figuring out how to do yarn2 on our dev machines + CI env + prod. So we just stuck with yarn1.
Now when Yarn 3 is out, it's either back to npm, which is supposedly on par with yarn1 in speed or I'll try a yarn 1 ->3 transition.
Even if you don’t have your build pipelines in the same or any repo, you could probably clone then modify them on a reasonable CI server.
But I don't actually want to _use_ yarn! More specifically I don't want to waffle my scripts, team, and keyboard habits btwn npm and yarn commands every time yarn has a new hot take on package management. Which is like, a lot.
- In practice, using PnP and removing node_module creates more problems than it solves. Disk storage is cheap, so we use Yarn with "nodeLinker: node-modules"
- Some "core" functionality like "yarn outdated" and "yarn install --production" is no longer available by default, and you need external plugins to mimic it
- Custom pre/post hooks in package.json are not supported. So if you have "prestart" or "postrelease", that will no longer run automatically.
But overall, it works well. We chose Yarn over NPM because we often use its "resolutions" feature that NPM has not yet implemented.
I haven't got the build working yet - and not sure it's even achievable - because so many of our explicit deps have implicit dependencies themselves (and some refuse to fix) - but it's been a valuable exercise.
I usually create a fork in those cases.
uses patched version of typescript. Using latest version of typescript is not reliable IME.
Bugs with npm scripts are really hard to notice, the terminal output is much more pretty, but imo has regressed.
Many npm scripts are not supported. You don't get a warning for which ones are ignored.
Simply viewing the output of your npm scripts is quite annoying, if you can figure it out.
Where yarn 1 "just worked" for my co-worker on windows (I'm on mac), yarn 2 did not "just work" - he got some security prompt (Yes, Corporate Sludge)
Our security scanner (blackduck) could not properly parse the newly formatted yarn.lock file, but, thankfully, I could create a work-aroung by telling yarn2 to use `yarn2.lock` instead... and just maintain 2 lock files. (Yes, more corporate Sludge)
Also, Svelte has a variety of issues with yarn2. In theory should be solved with yarn3.
I am hopeful for yarn3. May try again once it's been iterated on a bit. Maybe once yarn4 beta is out, yarn3 will be well worth trying again.
Biggest thing to me is the typescript situation. Yarn should be completely up-front about hacks like this. Yarn2 should immediately tell you to modify your package.json to point directly to the forked version of typescript when you first run yarn install, and in the migration guide.
zero installs is very neat, but, upload speeds are always slower than download speeds. Proper CI caching+not uploading truckloads to git will be the fastest+best strategy (as long as upload speeds are slower than download)
2. the npm lifecycle changes are very well laid out with many simple remedies see the docs
3. the list of deprecated lifecycle scripts are in the docs and very clear. the reasons are also interesting
4. yeah yarn requires you use process.stdout.write or write a yarn plugin
5. all of the issues you describe can be fixed via the sdk plugin
sounds like the corp stuff is the worst, though we’ve had similar issues with our security scans
I enjoy the simplicity of having a project level folder that has all the dependencies.
I like being able to easily expand it in my project tree and set a breakpoint in a dependency source file to debug strange issues.
For deployments, I like being able to see exactly what's being executed.
I would enjoy some more clear (to me) version locking though. I seem to constantly run into issues between environments with slight version changes, even though I commit my lockfile.
I've been using Yarn v2 on a decent-sized mixed-codebase JS app since the fall, and we just switched over to using it for both of the Redux Toolkit and React-Redux repos.
But to say that you eliminated node_modules and then introduce a new folder which will have the same files inside it... sounds like madness to me.
No thanks.
Conceptually, it's great. The problem comes when all the other tools in the ecosystem are still expecting a `node_modules` folder to exist and to be able to read those individual files directly, including Node itself.
From what I've seen, the compat story has improved over the last couple years, but there's still rough edges. Last time I tried it I ran into problems with things like import aliases, and of course VS Code and TypeScript and other tooling all had to be tweaked to work correctly with Yarn's filesystem overrides.
But, next time I start a new project from scratch, I'll probably give it another shot.
The problem of wasted disk space... start using a modern filesystem like ZFS (or BTRFS, but it's less stable) that does Copy on Write and you shouldn't have these problems. Or deal with it, nowadays disk is very cheap and fast so I don't see why worrying that much.
I live with npm and that is fine, it's the most compatible solution.
On system with high iops(ssd), the impact of file count is negligible.
On system with low iops (hhd, nfs or wsl on windows dirs), it is a nightmare.
I used to try to run npm install in an old project on /mnt/c/ on wsl1. And the install that takes 20~30 second normally end up took 20 minutes. Because /mnt/c/ under wsl has extremely low iops.
I don’t check in node_modules for two reasons: 1- It can be fully regenerated (at the cost of time) from a lockfile. 2 - It’s huge, and large git repos are tough to work with (and can get you a nastygram from github if you’re on a free plan).
Last time I tried yarn 2’s “zero installs mode”, the gist seemed to be “no more node_modules! Just check in this .yarn folder! Yes it’s technically the same thing, but we’ve always wanted to check in our deps and now we’re giving ourselves permission!” The .pnpthing file and .yarn (the parts that weren’t gitignored) added up to like, 40MB. That’s 100x larger than the rest of my project. Am I missing something here? Is this any better than just checking in node_modules? (Which would also technically get you a “zero installs mode”)
I love to use yarn workspaces but I have to wait for GitLab to fix the npm metadata issue first. But they have postponed it to October.
NPM now has npm ci (which I think should be the default) for that.
If so, ignore this, but if not, wouldn't adding a new package have all sorts of weird side effects from miscellaneous updates to irrelevant existing dependencies?
So it ends up with un-optimized structure sometimes. Because the top package is locked and does not meet the version new package required.
In order to satisfy the constraint. Npm will install second copy of the package in nested directory.
And npm also have a `dedup` command for you to `screw it, change the version in lock and merge them if possible`.
I think if you're careful about what you pull in from npm it's totally fine.
I also dont like that some packages are too big, or too small, or have too many transitive dependencies. It is a fact of life in npm land, though, making deterministic builds simultaneously minor and significant.
I just want Vercel to support it so I can use it to speed up build times with relying solely on build cache.
Also checking out branches and syncing takes all of 3s versus the massive link process we used to have to do.
Love yarnv2
Whereas Bundler just works (generally speaking).
V2 (now v3 ...) lives at https://github.com/yarnpkg/berry.
No idea why they did it that way. Probably because Yarn 1 is still very actively used, and things would get confusing otherwise.
Wish you the best with this and hope the tide turns on your favor eventually :)
Nepotism wasn't the reason the agency I was at kept starting projects with yarn (JS devs, whatcha gonna do) and pretty much always switching to NPM about the time each project was seriously getting going. It's because the likelihood of Yarn being the cause of some mysterious runtime or build bug you'd spend a couple hours chasing was at least 10x higher than NPM, or because we'd find we needed a feature Yarn didn't support.
Yarn proponents have a history of pushing it despite its not being ready. Perhaps it's actually good now, though, I dunno. NPM's definitely not perfect, so there's plenty of room to improve on it.