mhagemeister

This time we're going to look at the challenges and solutions to making JavaScript plugin support for Deno's Rust-based linter performant.

Author here. That's good feedback. Looks like that part of my article is a bit confusing and I've updated the phrasing a little to hopefully make it less so.

You're spot on with all of your points. The isolated declarations feature forces your code to be written in a way that creating the .d.ts files is a mere exercise of stripping syntax and can be done easily without invoking the tsc compiler.

For runtimes like Deno which support running TS natively (disclaimer: I work for Deno) you never had to care about creating .d.ts files when publishing your package to any of the Deno registries: previously /x, now JSR. In the background though we've always tried to feed the tsc compiler in Deno something like .d.ts files though as it's quicker for type checking purposes.

Author here. Agree! The whole npm ecosystem is based around npm packages shipping JS files and doing anything else would break it. I've updated the article to hopefully make it less confusing. The takeaway should definitely not to only publish TS files to npm.

I mainly work with Deno which can run TS files natively (disclaimer: I'm employed by them). This changes the parameters a little as all registries for Deno could always ships the TS sources directly. But when publishing to npm you always have to publish the JS sources to work with runtimes like Node that don't support running TS natively.

Author here. True, I think I should've worded it better. I work for Deno and it being a JS runtime that is able to run TS natively makes this less of an issue. The problem that a dependency needs a specific TS version hasn't come up so far. But that might be because Deno users tend to update to the latest version pretty frequently as we haven't done any breaking changes so far.

Author here. I should've worded that better. The recommendation is not to stop compiling TS->JS for npm users. The whole npm ecosystem is built around the assumption that you ship JS and doing anything else would break it.

In Deno (disclaimer: I work for Deno), which supports running TS natively, packaging outside of npm has always worked by shipping the TS files directly. Transpiling them down to JS files is only something you ever had to do when publishing on npm.

So far the problem that a dependency only compiles with a certain version of tsc hasn't popped up since Deno was launched. But that may have been because Deno always ships with latest TS and Deno users don't hesitate to update as we haven't done any breaking changes so far.

Oh no, I just woke up and saw this comment. It seems to be up again. Apologies for the inconvenience. Looks like the HN hug of death is real!

Happy to hear that you like the series!

Author here. I should've worded that better in the article. The takeaway should not be to publish only TS sources to npm. The whole npm ecosystem is based around the assumption that you ship .js files and doing anything else would break it. I've updated the wording to hopefully make it less confusing.

Author here. Unfortunately, I think worded that a bit confusingly in the article. I'm used to working with runtimes that run TS natively and didn't have to care about .d.ts files myself for a long time. From that perspective creating them by running the tsc compiler feels like a manual step for me.

There are cases where folks write them by hand, but that's not what I had in mind when writing the article. I've rephrased that part a bit to hopefully make it less confusing.

Author here. That's a good point and maybe a matter of perspective. I've worked on projects where using tsc to generate .d.ts files took close to an hour. Even 13s would be too long in my opinion. It should be instantaneously. I guess it depends on where one's personal tolerance threshold is. I have a bit of a lower one as I easily get distracted and loose flow.

Author here. Thanks for sharing that feedback. I think I missed the mark by being so used to working with runtimes that natively run TS files, that I forgot that this is not the default for everyone. The article was mostly written from that perspective.

For Node users the isolated declaration feature won't do much other than speed up the creation of the .d.ts files a little. For runtimes that can run TS natively like Deno (disclaimer: I work for Deno) the benefit is much more apparent as you always want the original TS sources, but can generate .d.ts files on the fly for faster type checking.

EDIT: Formatting

> I'd like tool to generate the whole package root, including a transpiled or generated package.json

That's exactly what JSR does. We generate the package.json ourselves.

I think you're looking for esm.sh . It transpiles both npm and jsr packages for the browser so that they can be used in a simple script tag.

The performance of type inference matters for runtimes which work with TypeScript files directly, rather then using the transpiled .js + .d.ts files like Deno. This has several benefits in that we can jump to the source on "go to definition" rather than some random .d.ts file among other things. Deno uses the original TS files for type checking as well and that's why inference also affects type checking performance in some runtimes.

By ensuring explicit return types in the public API, the generation of .d.ts files is turned into a mere syntax transform, rather than requiring the tsc compiler. This is something TypeScript will ship with in the next release itself. They're adding a new `isolatedDeclarations` option which is the same thing.

yup, we added the `jsr` tool mostly so that you don't have to be aware of the @jsr scope that's used under the hood. We're not publishing to npm though, but rather mapping the @jsr scope to the JSR registry in the project's .npmrc file.

We do some basic inference, but that's why we require explicit return types in the public API, see https://jsr.io/docs/about-slow-types . TypeScript itself will ship with a new `isolatedDeclarations` option in the next release which is the same thing. The point is that no inference other than simple literal inference needs to be done to get the type of the public API. That way generating .d.ts files can be done by any tool by merely looking at the syntax without type inference.

> This makes the already complicated module resolution logic in most bundler/packaging tools even more complicated as they must account for the intricacies of another package manager. A house of cards being stacked on another house of cards...

JSR is a registry, not a package manager. npm allows you to add multiple registries and JSR provides an npm API endpoint. From npm's point of view packages coming from JSR are the same as those coming from the npm registry itself. The only difference is where the package is coming from, but all semver resolution, etc are done as usual by whatever package manager you use.

Disclaimer: I wrote a good portion of the npm tarball generation code.

It's not a secret what jest does. There are a few abstractions to jump through, but I find jest's code very readable overall. By default the `workerThreads` option is set to `false`. Then we reach their `WorkerPool` class which pools the test runs. The "Worker" in the name is not to be confused with WebWorkers or node's worker_threads. It's merely the term the jest team has chosen. Inside that and because workerThreads is set to false, we instantiate a child process here

https://github.com/jestjs/jest/blob/00ef0ed0a03764f24ff568bc...

Author here.

Compilation isn't the problem, loading and instantiating the module graph is. The timings for that are currently not exposed in any runtime afaik.

Author here.

That's fair criticism. Whilst the setup shown in the article is synthetic, the experience is from working on real world projects. I've worked on a bunch of projects which had about 3000-6000 barrel files easily, and these then lead to every file in the project always being included. The total amount of files being loaded was somewhat around 30-60k, depending on the project.

I did the exact same thing as described in the article and wrote a custom test runner for jest that bundled the test files with esbuild before executing them. Even though bundling introduces a costly overhead on its own, it was still about 60% faster to bundle + run the bundled code vs not doing it. Bundling can also be amortized because it only needs to be done once per test run vs once per file. That's where the real gains came from, because in jest every single test file constructs the whole module graph from scratch.

I haven't checked myself, but there is likely improvements to be made in node (or other runtimes) to address this. Hopefully, some folks get inspired by this article to look at that.

They are unfortunately not removed, because the way they are used makes it difficult for bundlers to detect them.

Deno encourages you to submit the original sources which can be even in TypeScript if you want. The users are very close to the newest Deno release and there is barely anyone staying on old versions. This works because Deno takes semver very seriously, which in turn encourages folks to upgrade. It removes the need for polyfills and allows you to always use the latest JS features.

Disclaimer: I work at Deno

Glad to hear you like it! Those flame graph screenshots are taken from https://www.speedscope.app/ .

That's a very good point. Didn't know about the grammarly incident. I could definitely see this happening again with the amount of polyfills in npm packages. Polyfills are usually frozen in time and not developed further after they are released.

Thanks for the kind feedback! It's definitely something in the back of my mind. I feel like I need to collect a little more content to fill a whole book, but I'm enticed by the thought of writing one nonetheless.

Yeah, engines are a moving target. I'm all for backwards compatibility, but I'm worried about promoting old node versions with known unpatched security issues. Given that eslint itself only supports node >= 12.22.0 it seems like it's time to get rid of the polyfills.

I wish we as in the industry would find a better solution to adapt to this. It's a bit unfortunate that the polyfills as part of the library code itself, which makes it difficult to get rid of them once they're not needed anymore.

Author here.

Thanks for the kind words! It's feedback like this that encourages me to keep writing about it.

I share your experiences regarding babel helpers and haven't found a good solution myself. Similar to you, I often patch unnecessary stuff out via patch-package, but that approach doesn't scale well.

Author here.

Good point. Agree that the ideal scenario would be that the end user (or the tools they use) have the final say in which polyfills to load. It's a bit of a bummer that they are shipped as part of npm packages without an easy way to get rid of them.

I wonder if our industry will move to publishing the original source files to npm in the long run. Only the last piece of the chain, the developer using these dependencies, knows what their target environments are. So the bundler could then downlevel or polyfill the code for the specified targets.

Author here.

That mirrors my experience too on working in various projects. The automatic polyfilling story is such a good thing in theory, but reality isn't as rosy and much more polyfills than necessary are included.

Author here:

Thanks for the kind words! I'm in the same boat as you, profiling is a lot of fun! Yeah, the more you look, the more you realize how many projects have the same or similar problems. Hoping that more folks are aware of those issues and how to avoid them with this series.

Author here.

We decided to open the PR nonetheless as people kept asking us about it: https://github.com/estools/esquery/pull/134

Author here.

That's great feedback! After reading your comment and re-reading the section in the article it does indeed sound wrong. Decided to remove that paragraph. Your explanation of the string representation is really good. Thanks for sharing!