Comment by pornel | Hacker News Reader

Comment by pornel | Hacker News Reader

I think "npm bad" is a lazy take.

• There used to be a focus on attacking endpoints directly, but eventually OSes, browsers, and servers stopped being easy targets, so attackers looked for another way in.

• OS (especially Linux) distributors acted as gatekeepers. Getting into a distro requires having some reputation, and the risk of getting caught with a backdoor is a deterrent (although I don't think distros are as much of security barrier as people assume, since maintainers already have too much work to do and their source code review is at best at "LGTM" level).

• New tooling gave more visibility into dependencies. Transitive and build-time dependencies in precompiled libraries typically aren't visible, so it's easy to think you have one or two deps, but transitively there are still tons of them: https://wiki.alopex.li/LetsBeRealAboutDependencies

• Exploits became easier to monetize. Thanks to crypto miners and crypto wallets any exploit is almost a direct payout. Way easier and safer than trying to sell a botnet, adware or even stolen CC numbers, so it lowered barriers to entry to this "business" as well as made it more attractive.

I'm not sure "new tooling" (if you're contrasting language-specific with linux distro tooling) does give more visibility though. How well does "new tooling" track root certificates being up-to-date and available for example (ca-certificates is explicitly mentioned in the vlc dependencies in the link as an example)? Or across language/ecosystem boundaries? It would be interesting to compare vlc dependencies on Windows, MacOS and one of the BSDs, as unlike linux distros, there is a explicit and somewhat complete core (which would likely remove or at least reduce the number of "system" dependencies on the vlc list).

I do think better tooling can exist (and is being developed), but it needs to track the whole system, not just a single language.

Yes, I mean something like Cargo vs apt.

If you depend on libcurl, you count that as 1 (one) dependency. Nice, and restrained.

But curl has everything from gopher, FTP to SMTP and IMAP! It has a half a dozen TLS libraries, SSH, auth methods, multiple compression algorithms, socks proxies, etc, It's a lot of code, and a lot of transitive dependencies that you don't even think about.

OTOH when you add `reqwest` in Cargo, Cargo will visibly fetch everything needed to build it from scratch, and it will look like 140 dependencies, not 1. But in total it's still only a subset of what libcurl contains. There's no Samba or POP3. It's just a very detailed list showing you it contains mundane things like a CLI arg parser and base64 encoder, and a wrapper for ca-certificates too.

Please don't interpret it as a criticism of curl. It's just to contrast that you may see libcurl as 1 dependency, because it's one precompiled package, but you may see a smaller amount of code as extravagantly bloated shocking 140 dependencies, because it's just been split into tiny pieces and rubbed in your face.

Curl, even compiled against all of its optional dependencies to support every possible kind of auth, encryption scheme, compression, etc. is nowhere 140 dependencies, even transitively. All of Linux from Scratch is only 75 packages.

In contrast, when I tried to add a token-replacement library to mdbook, Cargo installed 187 crates.

There is a real and legitimate difference in the size and functionality scope of older C and C++ libraries with the type of single-function packages you see from NPM and Cargo that has made full supply-chain audits far more difficult. To get functionality equivalent to something like Boost, you might need to vet a thousand separate vendors, the majority of which are pseudonymous Github accounts with no physical address or backing organization.

I'm not just speculating about this, either. I used to be a platform lead on fully-disconnected, classified information systems where we needed to bring in and compile every dependency from source via DVDs hand-carried to a classified workstation. Somewhere around 2015 or so, this stopped being even a remotely feasible job and created multi-year backlogs of teams that had something fully working in their unclassified development environment with Internet access suddenly finding themselves staring at a minimum 6-month, more likely indefinite wait to get to production when they discovered security teams needed to individually approve every package and every developer they wanted to bring onto a classified system.

But does not cargo bottom out at libraries like https://crates.io/crates/openssl-sys/0.9.75, where you do need addition (non-managed) dependencies? I would be surprised that the more "conservative" linux distros didn't have a complete, deduplicated and unvendored tree (only encountering issues around binary firmware and similar), whereas from my minimal rust knowledge, cargo's reqwest metadata is incomplete. It's the incompleteness I'm suggesting is a problem (I'm more familiar with Python and wheels, where there are substantial non-Python dependencies people forget about), not the number of dependencies (though that's not great either).

The "system dependency" wrapper crates are unfortunately dependent on the environment outside of Cargo's view. The Rewrite-In-Rust Strike Force is working on eliminating this problem ;)

Cargo/crates.io has a publicly available index of every crate with every dependency. `cargo tree` lists everything in your project. `Cargo.lock` contains inventory of every package used in the build, including their checksums.

See https://lib.rs/stats

I agree with this.

The example of 1000s of packages having a transitive dep on a package which left-pads strings looks bad, however the two obvious alternatives to this are:

1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.

2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.

> 1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.

You sure? You can write it, with no security issues, with acceptable performance for most cases, in the same time than it takes to find the dependency. In fact, you'll save time since you don't have to vet the dependency.

Uh, you do like, vet your dependencies right, whenever you add them, right? As well as every time you update to a new version, right?

If you're not doing any of that and just adding dependencies willy nilly then I could see how it could seem easier, but this is an illusion. You need a dependency to do more than just left padding to make it worth adding, unless you don't give a crap about stability or security.

> 2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.

Yeah this is how languages have traditionally done it, and when done well it is very nice. As you say, it is hard to change the contract because it gets depended on, and you know what? This is a feature, not a bug. Things that are really simple and settled like leftpad don't need to constantly change. This is also why emerging languages try to stay "alpha" without guarantees about breaking changes as long as possible early on, so by the time they have to really freeze the standard library, a lot of experience has helped polish it.

A similar model is something like Java, which has done a fantastic job of evolving its standard library over the years while staying very (but, mostly, not excessively... mostly...) backwards compatible. Mostly through the trick of letting users create their own popular libraries and then eventually integrating it with the JEP process once one gets really big. Kind of like a lot of the stuff from Guava, or Jodatime.

3. Have a package of common utility functions maintained by organization rather than an individual, like Apache Commons Lang.

Reply on news.ycombinator.com