Supply chain attacks and backdoored dependencies
kerkour.com
kerkour.com
It seems to me that we hit an enormous growth phase in the last 10 years and invited upon ourselves an Eternal September of Bootcampers and other very green developers who can’t be trusted to do more than glue components together.
Convenient, centralized package managers make these folk way more effective at delivering products in the near term, but there are too few experienced leads around to point out the maintenance and security concerns that play out long-term when you’ve committed yourself to continuously folding in unreviewed code from 1000 unpaid contributors you never vetted and who know nothing about you.
After years of “screaming into the void” (as another commenter put it), I’ve accepted that this is reality and that there are reasons for it: it produces valuable work quickly, and remediates labor pressure by lowing the barrier to entry.
But I can’t shake that we’ve loaded ourselves on a train whose track runs right over a cliff and that the correction, if it happens before we fall off that cliff, will be very expensive and jarring.
As an aside, I'm not about to blame folks who came up through bootcamps. I've known too many outstanding devs from those backgrounds, and it's clearly enabled many solid devs to enter the field who would otherwise have been excluded. (I've also met a lot of folks with CS degrees who could give fck all about critical code infrastructure concerns.)
Likewise, for decades we've been striving to build languages, platforms, and tools that support powerful abstraction and code reuse. Somewhat amazingly to me, we've realized some of those goals. So I think the problem also isn't devs working at "glue code" level – at some point we must* work at higher abstractions to get useful work done. Reinventing the wheel (or $datastructure) gets old. Again, I'd return to noting that the foundations are rotten, and the age-old bugbear of complexity management has only gotten worse.
I both agree and disagree with this in some ways. I think the potential to escape is better than ever before. The concern is more about the culture and the people for me.
We have a very minimal vendor footprint with our B2B banking product (out of regulatory & market necessity). The only 3rd party software we depend upon aside from Microsoft's .NET framework stack is currently sqlite.
For web-facing properties, we mostly just roll vanilla javascript. The latest web standards are incredible and you are really doing yourself a disservice by hiding from what is sitting out on MDN right now. We don't even need to use crap like jquery anymore.
You are right about the "Eternal September of Bootcampers". Even guiding some curious friends down the path I was unable to keep them away from the dark alley peddlers of quick fixes and other ready-made software.
I know, it's not NodeJS/npm I'm talking about here.
npm/Node needs its own resource integrity type feature where we can cryptographically verify the integrity of packages against a known-good library of packages.
[0] https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
For CI/CD and some other cited attacks: again a student can be excused, a bit company MUST have it's own infra with it's internal operation, security etc. If that's not the case just because it's easy and cheap pay less internal technicians and talk more with colleagues commercials from some other companies whose jobs is just selling someone else technicians services with an automated UI they deserve the outcome.
Distracted devs who publish credentials, distracted admin who left default pig&dogs open instance of some services etc can always happen but if it's more than once in a while for a quickly fired employee than that means a complete IT department re-build is needed.
Please do not tell me the joke about the need of being quick and agile etc. Modern crapstructure are quick only in making disasters. To be really quick we need to came back in the history, restart completely erasing modern IT from IBM and onward and learn how to work properly, this, yes, is far from being quick, but that's the sole reasonable option I see. Keeping layering crap to offload crappy unsolvable problems to third parties like "cure the symptom > cure the the problem" is only a recipe for a worse disaster, like re-paint a wall with a broken corroded pipe behind: initially you quickly solve for few days, than for one day, than you haven't even the time to finish painting.
More real supply chain attacks, like some SoC who do not work as they say they do... There is only one mitigation: open hardware and internal production, the latter can't be done essentially by anyone, the former must be mandatory by law IMO.
A few other ways that attackers get write access to packages that were not covered:
1. Maintainer gives access to a bad actor. The most famous case happened in November 2018 when Dominic Tarr gave access to the package `event-stream` to a new maintainer who turned out to be an attacker. This bad actor offered to help maintain the package – something not uncommon in open source. At the time of compromise, event-stream had 1.5 million weekly downloads [1]
2. Maintainer goes rogue (i.e. becomes an attacker). The most famous case happened in January 2022 when a maintainer named Marak 'went rogue' and sabotaged his own packages `color` and `faker` which together received over 20 million weekly downloads [2]
I believe that the way forward is to assume all open source packages may be malicious. At socket.dev, we use "deep package inspection" to characterize the behavior of an open source package. By actually analyzing the package code, it's possible to detect when packages use security-relevant platform capabilities, such as the network, filesystem, or shell.
For instance, to detect if a package uses the network, socket.dev looks at whether fetch(), or Node's net, dgram, dns, http or https modules are used within the package or any of its dependencies. We can detect the tell-tale signs of a supply chain attack, including the introduction of install scripts, obfuscated code, high entropy strings, or usage of privileged APIs such as shell, filesystem, eval(), and environment variables. [3]
[1]: https://blog.npmjs.org/post/180565383195/details-about-the-e...
[2]: https://www.bleepingcomputer.com/news/security/dev-corrupts-...
[3]: https://socket.dev
Personally I'm very concerned about the lack of control over its highly opinionated behaviour, the intentional silent/background execution, the whole shimming concept, its opaque and unorthodox distribution, etc.
A supply chain is difficult to secure when your supplier starts things off by silently curl-piping an unverified yarn installation script in your userdir, instead of simply executing the `npm install` command you told it to run (e.g. because `package.json` lists yarn as a compatible package manager but not npm).
I feel like I am screaming into the void about this, and management is shrugging their shoulders when our entire house is on fire. Using NPM in 2022 without a package lock and depdency review is suicidal.
How does one guard against that? Review every patch?
Well, yeah.
If you review code contributed by somebody you spent hours vetting and interviewing, and whose worked in the next office over for a year, and whose work is accountable through their livelihood, why would you not review the code contributed by some unpaid internet rando?
Of course, the practical answer is “Because that’s crazy! There’s no budget for that!” — which is why this problem exists.
There are other ways of establishing and delegating trust so that you personally don’t have to review every import and change (see the Linux distribution ecosystem), but these vulnerabilities will persist until those networks and systems get built out.
And they won’t get built out until people learn that GitHub stars and popularity are not sufficient measures of quality or reliability.
Yes. You are responsible for the code that you ship to production, whether you wrote it or it comes from open source. Fundamentally, there's no other way to solve this problem in a dynamic language like JavaScript.
You can use tools like https://socket.dev (disclosure: I'm the founder) to automate finding the dependency updates that are particularly suspicious/risky, i.e. they contain the tell-tale signs of a supply chain attack, including the introduction of install scripts, obfuscated code, high entropy strings, or usage of privileged APIs such as shell, filesystem, eval(), and environment variables.
In the far future, I have hope that efforts like ES Realms and LavaMoat will give us per-package permissions and true isolation, but at the moment they're not practical/performant enough to use. Even with code signing (an often suggested approach), you still have the problem of maintainers going rogue, or maintainers adding new malicious maintainers to formerly safe dependencies.
tldr; you need to 'review' every patch, though it doesn't necessarily need to be a human review for every patch.