smarr

> is that Java compiled or interpreted?

The figure for the JIT-compiled numbers uses a standard HotSpot JVM, with JIT compilation. The figures for the interpreter numbers uses a standard HotSpot JVM with the -Xint flag, so, only using the Java bytecode interpreter.

The TruffleSOM interpreter is AOT-compiled, so, it's a native binary, which is then interpreting the SOM code.

With the amount of work that's needed to even attempt such a study and get to a point that one can precisely define which variable one is actually measuring, there are indeed compromises needed.

But please feel free to replicate the study for your preferred language. I am happy to discuss more about why we made certain choices.

Please see Figure 2 in the blog post https://stefan-marr.de/2023/10/ast-vs-bytecode-interpreters/... or Sec. 5.2 in the paper https://stefan-marr.de/papers/oopsla-larose-et-al-ast-vs-byt...

We report on results for both, with and without just-in-time compilation. The specific focus for this work was pure interpreter performance in the context of metacompilation systems, but before compilation had a chance to kick in.

For both RPython and Truffle/Graal, it's possible to disable the JIT compilers and measure pure interpreter speed.

Please see Figure 2 in the blog post https://stefan-marr.de/2023/10/ast-vs-bytecode-interpreters/... or Sec. 5.2 in the paper https://stefan-marr.de/papers/oopsla-larose-et-al-ast-vs-byt...

We report on results for both, with and without just-in-time compilation. The specific focus for this work was pure interpreter performance in the context of metacompilation systems, but before compilation had a chance to kick in.

You are free to disagree with the specific design choices of the AST and bytecode interpreters for SOM, but we put quite some effort in to have an as fair as possible comparison.

Yeah, the serve seems to be temporarily down. The paper can also be found here: https://tugawa.github.io/ejs-comlan-preprint.pdf

A few of the smaller benchmarks have been ported to C++ already: https://github.com/smarr/are-we-fast-yet/tree/wip/cpp/benchm...

But the bigger ones are not yet (so, no results yet).

One of the important questions to answer first is how the mapping should be done. A naive version using new/delete/smartpointers is going to have performance issues. Other options would be to use arena allocators and completely remove memory management overhead from the equation. Depending on what comparison/C++ usage scenario is desired, both options would be useful.

I tried Pandoc before reverting back to tex4ht. Unfortunately, it models a rather small subset of the things I was interested in. Specifically around the typesetting of citations and listings, as far a I remember. So, tex4ht and HTML post-processing it was.

It is built on top of tex4ht. It provides merely a few settings for tex4ht and post-processing scripts that beautify the generated HTML. You might ask why post-processing? Well, because it was simpler for me than figuring out how to get tex4ht to do the desired thing. I just find Tex/Latex not pleasant to use as a programming language, but that's personal taste.

Yes, that's indeed a none-obvious issue, and looks rather strange on the graph.

That's not the 'partial evaluation' per se. Instead, it is the difference between RPython and HotSpot that surfaces here. RPython does currently generated singled threaded VMs, and neither the GC nor the compilation are done in separate threads. The HotSpot JVM however can do those things in parallel and additionally has other infrastructure threads running. In the end, this increases the likelihood that the OS reschedules the application thread, which becomes visible as jitter.

Clean HTML (no superfluous tags), proper links to references (the rendering splits the links between author and year), ligatures are broken in browsers, standard footnotes are insensible for HTML, the standard HTML used for headers, figures, and others wasn't to my liking.

You are right, with a tracer, you get most of that for free. For a meta-object protocol, you'll need however still a little bit help to avoid creating too many guards. And, the PICs or `dispatch chains' are very useful in the interpreter, which avoids enormous warmup penalties.

That's a good summary, thanks. Glad that all that came across :)

Not to 100%. I used a combination for tex4ht, tex4ht configuration, several hacky post processing scripts, and manual editing. Converting a Latex document to a nice HTML site is easily the worst part of the whole writing process :-/

It is already used in the JRuby+Truffle backend. See for instance: http://www.chrisseaton.com/rubytruffle/pushing-pixels/

I don't follow that logic. First, I don't know what you mean with static. That you can't change the metaobject associated with a baselevel object? That's a loss of expressiveness.

And, what do you mean with runtime dispatch? Static and runtime dispatch combined? I am not sure I really know what you got in mind.

With dispatch chains, the runtime dispatch is resolved at runtime and the JIT compiler can inline through it. But, you need that runtime information. A static compiler, like for OpenC++ can't usually do that.

With generalized polymorphic inline caches (dispatch chains) you get that cost down to the dynamic check and the JIT compiler can remove all reflective overhead.

So, in the end, you get a powerful MOP without reflective overhead.

There is compile-time metaprogramming, staged programming, and many other things. Yes. Indeed the blog post neglects to go into detail about those for brevity. But, the point of the blog post was after all to get you guys to read the actual paper. Perhaps that wasn't explicit enough. Sorry.

The main point of the paper is to show that runtime metaprogramming can be optimized with a classic JIT-compilation technique. After one sees it, it's kind of trivial, but it isn't done in common VMs.

RPython is certainly not perfect, and the PyPy/RPython community is also very much aware of RPython being something that developed over time. But, RPython was from the start 'designed' as a language for implementing dynamic languages. The problem is that it took multiple iterations to identify meta-tracing as a foundation that works. And those iterations left some cruft here and there.

Compared to the Java-based experiments I did, sometimes I miss the ability to be explicit about types, and have to use RPython's assertions instead, which is a clumsy way of telling RPython about types. But overall, RPython isn't the worst toolchain I have ever worked with. And, typically, if the error messages are really bad, the community is very helpful and eager to improve them.

The one issue you seem to consider as not relevant at all is the effort it takes to implement a 'fast enough' language. With RPython as well as with Truffle+Graal, you can implement a language in less than 10k lines of code and get performance within reach of state of the art VMs.

Sure, if you prefer more static languages, you can probably move the point of optimization from runtime to compile time. However, I would like to see that you can achieve the same degree of performance by using LLVM for a small language also in the range of 5k-10k LOC. I am not aware of any similar experiments in that field. Rust, Go, or Julia seem to be all larger and more complex languages, so a direct comparison isn't really fair. Would be interested if there is something out there I have missed so far.