Hacker News Reader

Comment by Atotalnoob | Hacker News Reader

Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?

View on HN

For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results.

It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "warm" (and even then with scalable containers that assumption may not even be true in some cases). For servers, however, what matters is mostly how fast its HTTP library is and how good the async IO is... check Techempower benchmarks for that: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

From nothing to result rarely happens in real live. I hardly see someone to start/stop a program per unit task (like piping commands).

Code compilation is a common example. In many languages, when you do a clean build of a large project, you run a large number of short-lived processes.

With command line tools, the dominant design philosophy is that a program completes one task and exits. The tasks may be large or small, and the same tools are often expected to handle both.

Some language implementations have run time startup costs (bytecode verifier), not-required at run time by other language implementations.

https://www.oracle.com/java/technologies/security-in-java.ht...

Why should that run time startup cost be ignored?

Yes, but the author claims the longest JIT warmup is 0.3 seconds, so it's not an important issue in these benchmarks that take several seconds.

I strongly suspect that the author may have confused the JIT warmup (hard to measure, as you need to ensure that the performance figure have reached the stable point) from the startup overhead (easy to measure).

...and many jitted vms do not even reach a stable point at all, there was a big paper on this a couple of years ago.

This?

https://tratt.net/laurie/blog/2018/why_arent_more_users_more...

Yes, this is the one. Time flies! And VM jits didn't change all that much.

0.3s is significant for a task that only takes 1.14s

We don't know that OpenJDK was "Some JIT-based language runtimes take up to ~0.3 second to compile and warm-up"

Maybe that was PyPy with program times >5 seconds.

What's CLI?

Presumably command line interface/terminal. I can type `dotnet run` in the terminal and provide some options to runa .NET program for example.

Reply on news.ycombinator.com