Hacker News Reader

Comment by cercatrova | Hacker News Reader

As I mentioned in another comment [0], compiled languages are the future of interpreted languages' tooling.

> Whereas before one might have thought that a language's tooling should be written in said language, there might come a time when that's too slow, and we instead need to move to faster languages.

> That's what has happened to Python (numpy, pandas, scipy etc are all written in C and simply provide an interface in Python), and now it's happening to Javascript as well, with Deno, swc in Rust, Bun in Zig, esbuild in Go, and so on.

[0] https://news.ycombinator.com/item?id=32577837

View on HN

Hmm, maybe this is a dumb question, but Numpy, Pandas, Scipy, etc are all libraries. Are libraries considered tooling? When I think of tooling, I think of linters, debuggers, IDEs, and so on.

I can definitely see why libraries should be written in faster languages, they are used all over the place. And anyway, something Numpy is most importantly an interface to pre-existing high performance libraries (BLAS, LAPACK) which were already written in faster languages.

I think it is less obvious that a tooling in the second sense should be written in a lower level language. There's the obvious flexibility tradeoff, the fact that a person interested in a linter for a language probably is most familiar with that language.

I'm not familiar with the computational problems in linters. Is getting better CPU performance a huge concern?

A linter like Prettier can take several seconds in large codebases while the one by Rome (rome.tools) takes fractions of a second, because the former is in JS and the latter is in Rust. If I'm a linter user, I really don't care what the linter is written in, because I'm never going to open up the source code to look at it. It likely exposes a configuration file which is agnostic to any programming language, and that is what I will primarily be editing.

Same thing with Webpack vs esbuild or swc, the latter two are simply instantaneous while Webpack can take minutes, even, to build an app. That it's in JS does not matter to me since I don't care about Webpack source code, I only care about what the tool itself does.

Is a compiler "toolchain" package tooling?

Is the standard library included with that "toolchain" tooling, or a library?

So, I don't think it is really necessary for there to be a perfectly bright line between the two. But for the sake of argument, I'll say that my opinion is that the compiler, linker, etc are part of the tooling, while the standard library is a library.

All of the libraries are linked to the project by some tooling, and a particular distribution of software could include any number of libraries (e.g., conda), I don't see why the standard library needs a special cutout.

It will get even more blurry if you start asking about runtimes, macros, and all-template libraries.

Often (parts of) the C standard library (those needed for a freestanding environment) are actually "baked in" to the compiler, and not actually separate header files at all (though equivalent headers may be provided as well).

In C89, those are float.h, limits.h, iso646.h, stdarg.h, and stddef.h.

C99 added stdbool.h, and stdint.h.

C11 added stdalign.h, and stdnoreturn.h.

So those headers are part of the standard library, but they're also usually part of the compiler. The line gets very blurry.

Is that a bad thing? Libraries like Tensorflow were never going to be written natively in Python, but it's a reasonable choice for deployment (whls can be installed anywhere), for user interface (Python can interop with anything), and for overall ergonomics/ecosystem (import antigravity).

No, I'm arguing that it's a good thing. There are some who argue that a language's tooling should be written in said language, but I argue that that's putting the cart before the horse, that the tooling should be fast and efficient first and extensible second, because there are many more who are tool users than tool creators, so we should optimize for the former. There is also no reason that someone who is so interested in tooling could not also learn the tool's language, like Rust, and contribute that way.

I guess there's kind of a vague philosophical argument to be made for dogfooding, but that's perhaps more relevant early on in the lifecycle of an ecosystem.

On the other hand, if Python had been really committed to dogfooding at the level of something like golang, then the primary implementation would look a lot more like pypy than it would like cpython.

A counterpoint is the typescript compiler, arguably one of the most complex pieces of dynamic language tooling out there. It's written in typescript itself, and reasonably fast given that it has to implement a hugely sophisticated type system.

This shows that it's entirely possible to write fast tooling in dynamic language if you know how to.

Is the compiler expected to run only where a JS interpreter exists? (Which, at the time TS was created, consisted solely of Node?)

Python is kind of a special case. It's slow, widely used and often used as "glue" calling out to other programs and libraries that do the real work. There's also a lot of overlap for python and systems programmers, so there's tooling and that community knows how to do these things. There's more of a culture in python that to do anything serious (read fast) you write that part somewhere else and call into it.

If you look at JS tooling like eslint, tslint, uglifyjs, tsc, webpack, ~~esbuild~~, etc, it's all JavaScript (moving to typescript if anything).

EDIT: I stand corrected in that esbuild is not written in Javascript, but I think that's a disadvantage to it's success if anything and would point out webpack is more popular

As I mentioned above, new JS tooling is beginning to be written in non-JS/TS languages. esbuild which you mention specifically is in Go, not JS.

esbuild is written in Go, which is a large part of why it’s way faster than other dog slow tools especially webpack in your list.

> EDIT: I stand corrected in that esbuild is not written in Javascript, but I think that's a disadvantage to it's success if anything and would point out webpack is more popular.

Webpack exists since 2012, of course it's more popular. For a long time there were no other alternatives, so people had to deal with its madness.

Honestly I believe the low level core, dynamic shell is as old as programming. It's a bit part of the make it right then make it fast.

As I learned during my Tcl days, languages like Python should stay as scripting languages, because as you pretty well describe, you end up writing everything in C anyway.

A lesson learned 20 years ago, before everything .com went bust.

I agree, but I do think the big question will be supporting plugins without recompiling. The answer is probably WebAssembly or one of the native-node interops but it's still not trivial.

I’ve wondered how much those speedups are due to Rust/Go/Zig/etc, versus how much is due to being able to design the system with fresh eyes and an eye for performance. JS is almost surely slower than Rust, for instance, but it’s not 10-100x in most cases, at least in this random set of benchmarks I found [0].

0: https://programming-language-benchmarks.vercel.app/rust-vs-j...

Benchmarks are often not a great indicator for real world performance, especially for Javascript.

Benchmarks are very constrained, repetitive and often numerical code that is executed a lot of times with similar input. This gives the JIT a lot of opportunity to optimise and specialise for specific types.

These tools are usually not long-running , so code has to be JITed all over again on each invocation. A lot of code will probably only be executed a few times, so might only be executed by the interpreter or low tier JIT.

There will also be a lot of AST construction, walking and modification, which is really difficult to optimise without the guidance of types. A lower level language can use better data structures that don't require so much pointer chasing. Also lots of potential GC pressure if there are multiple internal representations to construct.

JS can be insanely fast for what it is, but alot of the weaknesses are apparent with short-running tooling. There is also a reason why most complex,JS heavy web apps seem to get laggy and slow eventually...

A major benefit of Rust/Go/Zig is control over data layout. If your language doesn't have value types, the CPU spends a lot of time pointer chasing and trashes the cache. In the worst case a cache miss could have the CPU sitting idle for hundreds of clock cycles while it waits for a RAM fetch to finish. That has a big effect in large programs with many layers of data structures, but it's not usually captured in microbenchmarks.

As already mentioned in neighboring comment, these languages give you control over data layout. But what's really powerful - is that control over layout opens up a possibility of using SIMD instructions, which by itself may have 10x speed up over optimized scalar code in the same languages, speed up over JS is, of course, greater.

Reply on news.ycombinator.com