Fnl: zero-overhead Lisp syntax for Lua
github.com
github.com
The largest reason for use of S exprs is making Meta programming easier. While certain things like arithmetic are more difficult with S-exprs, Macros and DSLs are far easier than with C like syntax. Since Lua is actually only a few steps from something like Guile, it takes very little code to translate a scheme like language to Lua without adding too much runtime overhead. The difficult parts of lisp to implement, namely closures, come for free.
With LuaJIT as good as it is, I would not be surprised if something like this would outperform guile.
LuaJIT is most certainly faster than guile, but it isn't a very good comparison. Guile is far from the fastest lisp (or even scheme, for that matter), whereas LuaJIT is probably the best JIT on the planet.
As for use-cases for Fnl, I would say it's more applicable for places where one would use Lua or Guile, rather than something like SBCL or any really fast, compiled Lisp.
Lua makes a pretty good dynamic language target, though, because of the many great implementations and small language spec.
LuaJIT is pretty friggin great and I have embedded it a couple of times with great success :)
I do however preferred guile model for multithreading with one instance that an spawn multiple threads, and that can be used safely from multiple threads,but that is just amatter of taste. If you want to spin up multiple instances, guile isn't for you (although you could use multiple threads in a single instance).
So whatever floats your boat :) There was some work implementing lua on the guile VM, which would have been cool.
Where can I learn more about how LuaJIT compares to other JITers?
http://lambda-the-ultimate.org/node/3851LuaJIT also does: constant folding, constant propagation, copy propagation, algebraic simplifications, reassociation, common-subexpression elimination, alias analysis, load-forwarding, store-forwarding, dead-store elimination, store sinking, scalar replacement of aggregates, scalar-evolution analysis, narrowing, specialization, loop inversion, dead-code elimination, reverse-linear-scan register allocation with a blended cost-model, register hinting, register renaming, memory operand fusion.
Due to the nature of a trace compiler, it implicitly performs partial and interprocedural variants of all of them. And many traditional optimizations, like straightening or unreachable code elimination are unnecessary.
All of that in 120KB for the VM and 80KB for the JIT compiler. And I didn't need 15 years and a billion dollar budget for that, either.
I'm planning to add value-range propagation, array-bounds-check elimination, escape analysis, allocation sinking, if conversion, hyperblock scheduling and auto-vectorization. Anything I forgot? I'll see what I can do. :-)
Implementation of those slightly diverges. Macsyma internally is largely written in Lisp syntax, where for example Reduce is written in RLisp (which is a Lisp written on top of Portable Standard Lisp), which does not use s-expression syntax.
When we all started coding, we probably used a text editor and thought in terms of editing characters: backspacing, typing, maybe mousing around for bigger changes.
Then at some point we graduated to a programmers editor where we got comfortable cutting, pulling, yanking, killing larger units of text: words, lines, paragraphs.
With tools like paredit, you can take another evolutionary step to edit the structures of your code. I find this very productive. It feels a little like those code block programming languages for kids.
If for no other reason than to enable slurping and barfing code structures in my text editor, this is a win. May all languages someday have a lisp syntax.
Imagine not being able to save syntactically incorrect code, or being able to have semantic diffs between domain objects (e.g. functions, classes, strings, types, apis) rather than thinking in terms of characters and lines, or being able to query a code base, like "find me all places where I assign to a variable named 'status' with type under 'error' and list me the names of the functions whose frame the variable belongs to". Paredit is simply where text and ast manipulation converge to the closest point.
If your development tool knows how to handle a particular language, then it can handle the AST for that language, but not any other. You could instruct the tool about the other languages, but that requires ways to let the tool know; in the end this is similar to the static versus dynamic typing discussion: static typing doesn't allow any potentially unsafe data, and a tool working with an AST directly wouldn't allow any potentially invalid programs, which in principle (or theory) is nice but difficult to achieve practically, it presents overheads for the tool implementer and for the user probably alike. And it probably takes away a lot of interactivity: just like reloading modules in a statically typed language (like ML or Haskell) is not as direct as being able to modify values and functions in a running system and having them be called directly.
So I think S-expressions are a very pragmatic concept that is going to be difficult to replace without the replacement costing more than it benefits.
S-expressions are a syntax for data. It's more related to what a tokenizer does, but with nested lists of tokens - not a flat token stream. It's only loosely related to an AST, since the s-expression represents some form of nesting - but not explicitly what kinds.
S-expressions know nothing about the syntax of the program language Lisp (what is the syntax for the LET operator?). If we have a Lisp expression, an AST for it would need to represent specific language constructs - like knowing that LET is a built-in operator and has parts like a list of bindings (where each binding is a symbol or a list of symbol + value form, optionally declarations (with their specific syntax) and a list of body forms. The s-expression does not represent that: in
(let ((let 'let))
(list let 'let))
Writing Lisp code using s-expressions allows us to use data manipulation functions both on text and on internal Lisp data: reverse, transpose, move, move in, move out, ...
We can then also enrich the editor (or other tools) to understand the syntax of Lisp. Example: know what the structure of a LET form is and then allow for example to manipulate the binding list, knowing where it is and what a variable and what a value form is.
What about Parinfer? I don't use it but seems like it's even lower friction than Paredit.
Going deeper, it would be nice to not being able to enter syntactically incorrect code.
We do see small examples of this in GUI's with strict validation on inputs that is applied for each keystroke.
One useful compromise is syntax highlighting that doesn't let you enter syntactically incorrect code without flagging the problem in a loud red color.
pr
pr;
printf(
printf(<?expr?>);
if␣
if (<?condition?>) {
<?body?>
} Assign(targets=[Name(id='x', ctx=Store())], value=Num(n=5))
A Lisp compiler might build up something like that, from a fully macro-expanded body of code. That's not usually accessible to programs.
Like the fact that n is converted to a Name() node, where it has a ctx property indicating Store() has nothing to do with syntax; it is the result of an analysis arising from how x is being used semantically. The user didn't specify any such attributes on the x just that x is assigned and that is already recorded by something like Assign(x, 5).
The whole premise around Lisp is precisely to express higher level concepts with lesser boilerplate.
This is achieved through a syntax that is really close to the AST, and macros.
The data structure representing Lisp data (i.e. code) is stripped of relevant text editing attributes, like how it is divided into lines and using what indentation, and, oh those semicolon-delimited comments.
Text editors also have to let the user deal with bad syntax.
So it's not just a matter of, "oh, we're written in Lisp, so just hand the buffer to the Lisp reader, do something and spit it out again".
Take Java for instance.
In Eclipse the a lot of AST-aware editing features are hidden in the "quickfixes" which you can trigger with CTRL+1.
For example you want to use an ArrayList but you haven't imported it yet. Use quickfixes: Bang the import was created with just two keypresses.
In a raw text editor you have to remember the full package string or google it but when you want to look up what a "XYZ" it's rather easy to find it.
The java import system is verbose but it makes it obvious what you're importing.
one class = one file Want to know what X is? Just open X.java.
Compare it to something like C/C++ where you include a header that contains multiple definitions. I honestly cannot use C++ without an IDE because of this. At the bare minimum my editor needs to be able to jump from a function call to the function definition or variable declaration to the struct definition.
In this case a feature that reduces boilerplate actually made the language harder to use without AST aware navigation.
Then there are quality of life things like extracting parts of a big function into seperate smaller functions. Just select the code you want and the IDE is going to create a function definition based on it and replace your selection with a call to that function.
I don't see how you could avoid this by making a more expressive language.
Does your fancy programming language automatically organise your code into functions or data types and all you need to do is crank out code without a care? No, you still to need think about that yourself, but the IDE can automatically move the code around for you.
Also let's talk about my favourite feature that I would never use without the assistance of an IDE: typeinference.
value blah = foo.bar([1,2,3])
What the hell is a blah? With an IDE I could just hover over blah and instantly know what the function returns. Without that feature I have to go to the definition of bar which is still easier with an IDE. After all if foo is also defined via type inference I have to lookup several function definitions and that's going to take some time. On top of that a tool like grep is imprecise and will give me hundreds of call sites but only one definition.
You can't fix everything by making a programming language more expressive and in my experience the dumber a programming language is, the easier it is to use with a dumb text editor.
Paredit is just a nice-to-have. Many lispers don't use it.