Ask HN: How to design an array lang (ex: APL or J) LLVM compiler?
Are the standard compiler books all still relevant here? Anything else one should know?
You can read about J here:
https://sblom.github.io/openj-core/ioj.htm
(From my long list of resources: https://gist.githubusercontent.com/mamcx/e1743571b9a1ea163a7...)
The main gist is understand that you can generaliza arrays and scalars in one single structure (aka a array!)
A scalar is a array of 1. Others list are of N. You tag the structure with the type alike:
[Type; N, N2, N3...]
for i in 1:
I expand the idea to say that a relation is a multiple-array (aka: Columns), and array is a relation of 1 column, a scalar is a relation of 1 column and 1 value.
This mean is potentially easy to exploit SIMD and stuff like that, but I haven't explored that.
If interested, we can share ideas?
My web with contact details: http://elmalabarista.com
I've never implemented a array language myself, but one way of doing this while minimizing cognitive overhead would be to lower it to more C-like IRs (Similar to how Haskell is compiled by GHC). The advantage here, would be that by the time the AST has been lowered into these IRs translating directly to LLVM IR would be mentally trivial. Whether this is worth doing or not, I have no idea.
I'd have a look at some LLVM APL compilers and see how they do it.
The J Incunabulum provides a glimpse of a toy interpreter and of Arthur Whitney's style: http://code.jsoftware.com/wiki/Essays/Incunabulum
I've found it quite interesting in terms of data structures, parsing, evaluation and how the primitives are implemented. I reformatted it into a more readable style (more readable to me). Memory management is through reference counting I think (maybe not in this example, but I vaguely remember reading that somewhere).
Interpreters for array languages are quite efficient because the primitives operate on arrays and you tend to not have explicit loops in the scripts themselves. The loops are "hidden" inside the optimized primitives in the interpreter. You can add further optimizations by recognizing certain idioms like +/ for "sum" and providing a special optimized implementation. Obviously that doesn't work for user-defined functions. I don't know if K etc can optimize those as well.
Another thing to consider is how you represent intermediate values. If you're operating on large arrays, you don't want to create a lot of large intermediate arrays that you'll throw away immediately. You can probably detect situations where you can modify arrays in-place.
This papers provides as different solution: http://dl.acm.org/citation.cfm?id=512761 Unfortunately I don't know if there's a free copy available anywhere, so I haven't actually read the paper. I'd be interested to know if there's a free legal copy available somewhere.
It might be similar to stream fusion for lists in Haskell: http://fun.cs.tufts.edu/stream-fusion.pdf Or how you might optimize SQL/relational queries for databases by pushing down projections through selects and joins to reduce the amount of data that you need to retrieve and work on.
These approaches might also allow you to optimize user-defined functions.
Finally, Timothy Budd implemented a compiler for a slightly modified version of APL and documented the result: http://www.springer.com/de/book/9780387966434 Again, I don't know if there's a free legal PDF anywhere, so I haven't read the book. I'd be interested in a free legal copy of this book as well.
You could compile to C, then use Clang or GCC to compile to a final executable: This should, if possible, be avoided as it limits what you can (C is not a good IR for a compiler). I suspect this is what you meant by "use regular C as the VM".
As these things go, LLVM is fairly sane/clean. The point of using LLVM would be to produce an optimized executable, rather than bytecode to be run on a VM.
Also, LLVM is C++ (Complex C++, in fact). If you don't already, I thoroughly recommend reading about optimizing compilers work (It's either very useful or fascinating, depending on whether you're the kind of person to find this stuff interesting). Using LLVM is, after getting over the C++ build times, actually quite simple (Take a look at http://llvm.org/docs/tutorial/)
It's in a VERY alien dialect of C, but quite readable with some effort:
To that end, it may be interesting to study the source of J, Klong, Kona, etc. There are a lot of older papers about representation and performance in APL as well.
Aaron's Co-dfns[1] is an APL compiler that compiles to C++ and there's some advantage to doing this.
It compiles to C++.
> The other problem is you have to have Dyalog APL, which ain't cheap.
Good isn't.
However, you can download Dyalog APL for free and use it for noncommercial purposes, including trying it out and personally learning how to build an APL compiler.
I think this is pretty clear from the website.
> I think a new array language that has a REPL that JITs to LLVM (good for prototyping, debugging, and testing) and AOT compilation to LLVM for scripts would be amazing for sharing code and deployment.
Why do you think that?
I don't see how LLVM helps with any of that. The hard part of writing an APL compiler is parsing and analysis, and that means finding an algebraic morphism between APL and CPUs (or GPUs) that you can do meaningful things with. It isn't making executables.
http://www.compileroptimizations.com/category/loop_fusion.ht...
https://www.google.com/search?q=array+language+loop+fusion+f...