Understanding the Power of Lisp (2020)
joshbradley.me
joshbradley.me
Still remember all the meetings that generally always included someone complaining "what the $#&^%! were they thinking and why didn't their Management stop it before it got this far."
The programmers were based in Canada and only wrote in Haskell. The CTO of White Ops had apparently hired them previously as contractors at a prior company without issue.
In hindsight, it turned out the purpose of Haskell was specifically because it was very difficult to hire anyone to replace them, and they wrote in a strange manner that left even experienced Haskell developers confused, especially without their customized IDE extensions.
After a couple rounds of funding, the Marxists went on strike and demanded huge amounts of equity from White Ops to continue working. Ultimately the company wound up rewriting everything in a more popular language than acquiesce to their demands. I heard the Marxist group also had tons of drama because the leader made everyone live onsite in shared housing.
There was a furry who worked as a web developer for a small firm. He had built their back-end runtime in MUCK scripting language. (A MUCK is like a MUD, but without the statistics and stuff, so it's much less a role-playing game than it is simply role-playing.) Eventually he was fired, probably because he spent more time pretending to be a highly sexualized female fox on MUCKs than doing his job, but the back end to all their web apps was written in MUCK code -- and few non-furries even knew what a MUCK was let alone how to program one. So the befuddled engineers who remained had to figure out how to modify the code without their vulpine-in-his-own-mind former colleague.
They uh, also didn't notice the backdoor he put into the web server that allowed him to spy on their activities, even though he was gone from the company...
I think i know these guys! Their names are Karl, Vlad, and Fred, right? Last I heard they founded the International Common-Lisp Party
In parts of the community we do not so much think of the tools as part of the identity of the engineers. We’re looking for general intelligence/capability and train specific technologies on the job.
- A desktop application for a bank in which not only the gui’s content but the look and structure was generated or changed on the fly mimicking the status of a database.
- An interpreter in which most of its classes and methods were automatically generated. The Java rewrite took pretty nasty code generation hacks, visitor pattern madness and way more LOC.
- Some program with a huge cache (glorified hash table) that was not catching simple strings or objects as keys but actual code as s-expressions. I always thought that was pretty cool.
Most if not all of my Lisp (and Perl) projects have been eventually rewritten in Java, and not exactly due to technical issues. I mostly put up with Go these days which is great most of the time but of course makes me yearn for a more powerful language or wished I never dabbled with those.
As for "just using AST", it is tedious to me, that's why I don't bother with it in other languages, and judging by the code that I've seen, it is for the average programmer as well.
And a college of mine took the application and easily made a version for another customer in less than a day. Changing most of the 52 pages to match the other customer. The look, the page layout, some of the logic etc. He never looked at the source code.
And mind you I have written my own Lisp dialects just for fun, and have written code in Common Lisp and Scheme. I like the simplicity of Lisp the same way that I like the simplicity of Forth. But I just prefer other languages. Not having types is tedious to me.
Some time ago I wrote a sort of DevOps tool in Python (because I know better than to start new projects in Lisp in most settings). This tool had preconfigured tasks defined as point-free style functions that were easily composable. Non programmers were able to define their own pipelines either submitting a YAML file or dragging & dropping these components in a web UI. This was very powerful, but anything remotely complex (if you didn't want to end up reinventing Forth) required dropping down to Python to define these.
JSON and YAML are very good at conveying structure, but fall sort for anything else. It would be like doing any sort of complex programming with Ansible via just using roles and playbooks. It's possible, but no one sane does.
Now the question is if this really matters. Judging at the shape of the industry, the clear answer is no.
And how close we came... and yet so far =(
We were working on a DoD project to feed relatively large amounts of logistics data in and out of a bunch of simulation models written for different contracts for different branches of the military. The inputs/outputs were all relatively large/complex files, with different formats and different units, and some values had to be repopulated in the output-files before being fed into the next simulation, and the input/output formats of many of these things were often being changed. The whole environment we lived in was more dynamic than you'd ever really want, yet I remember it feeling really easy to describe what I wanted to do in code; more than any language I've worked in since, stuff tended to Just Work the first time.
Some of the cooler things one could do in LISP that were difficult/impossible to do in other languages:
- Running in an interpreter from your editor (emacs, of course) allowed you to write editor code that could read/rearrange/fix your application code in the running image making the tweak/reload/test cycle trivially fast, even in really complicated systems (Python comes close, but not quite there).
- If it made sense, you could write lisp code in your editor that helped you write/manage the LISP code for your project, create patches automatically, etc.
- Writing a refactoring script in your editor was often better than trying to do it by hand. Decades later, I have yet to see an IDE or a refactoring tool that works as cleanly.
- The LISP macro system made it "easy"[1] to avoid writing boilerplate code. It was often more practical to write a boilerplate-generator, rather than thousands (on 10k) lines of boilerplate.
- Since classes/objects could be redefined on the fly almost arbitrarily, it was possible to redefine object data members as static class-members, and subclasses with overridden member-data, depending on the contents of your dataset AFTER loading it (at the time, I think this had a RAM savings of ~30%?)
- CLOS is the cleanest, most flexible object-oriented language I've ever seen, with a sensible multiple-inheritance system and all sorts of tools (like "before", "after", and "around" methods) you could use if you had a good reason to.
- The metaobject stuff in LISP is deep, dark magic; but sometimes you actually need deep, dark magic. Not often, but sometimes.
In the end, I think LISP is still a great choice for really complicated, dynamic problems (the kind that take years and millions of lines of code to solve). If you find yourself working on the sort of problem that makes you feel a need to build a domain-specific language to solve it, that's the sort of problem that LISP is REALLY good for. While I'm not a manufacturing engineer, LISP feels a bit like the programming language equivalent of a CNC mill/lathe that you can use to build out a factory that might require building bigger/better CNC mill/lathes.
[1] "easy" in that the non-boilerplate code was easy to read and understand its intent, but often it was much harder to debug, as macros often called macros which called macros. Debugging code that writes code is a whole different level of brain-hurt, but sometimes it's still better than the alternative.
Out of interest where are you working such that you're using Lisp? Pretty sure I've actually asked you before ha, but can't remember. Every-time I get burned out with frameworks and polygot development I end up daydreaming of using a stable language like common lisp for the remaining ~40 years of my career.
I am deeply bound up in Common Lisp because of my history. However, if I may make a suggestion: you might want to take a careful look at Racket: easy to make standalone apps, good libraries, large community, and if you don't like Emacs, the Racket IDE is really pretty good. Sit in on one of the online Racket conferences to see many kinds of cool applications and libraries people use Racket for.
• types let me play with the architecture and interfaces top-down
• the REPL lets me write more complex logic bottom-up
Usually there are a few "hotspots" in the codebase that require a lot of logic and are easier to develop bottom-up; the rest of the codebase is focused more on organization, domain modeling and information flow, where top-down design works really well. In actual practice I expect my mode tends to vary even more than that, but this division seems like a good approximation of how I think about it.
When I write Lisp I usually start testing in the REPL, iterate until the test code gets to be a few lines long, and then copy it over to the package's unit tests. Then I can keep iterating on both the test code and the code being tested.
I'll do something like this in the REPL:
(ql:quickload :mylibrary.test) (mylibrary.test:my-test-function)
And the REPL is also really great for "what if" testing that you may not want to keep forever, like, "What happens if I call this function with an invalid parameter?" Hopefully there's a test for that, but it's often a lot faster to just call the function and see what happens rather than go dig through the test code and lose your focus.
C++17 made this minimally useful, and things got spectacularly better with C++20 and constexpr vector/string. 23 is shaping up to move even more of the standard library to constexpr.
(cdr '(x a))
; does not return a, it returns (a)
(cdr '((x a) y))
; does not return y, it returns (y)
(cdr '((x a) (y b)))
; does not return (y b), it returns ((y b)) ((ab . c) . d . nil)
((ab . c) d . nil)
(eq '(a b) '(a b))
; (a b) is a list and cannot be evaluated by eq
A couple more examples for the Common Lispers in the audience that highlight the difference between the reader and the evaluator:
(setf foo (list #1=(list 3 4 5) #1#))
(eq (car foo) (cadr foo)) ; --> ??
Answer: It might return true in some implementations (very weird implementations), but in general one should not expect it to do so. Why?
(setf foo (list #1=(quote (3 4 5)) #1#))
(eq (car foo) (cadr foo)) ; --> ??
Answer: It absolutely must return true. Why?
One of those is called SBCL.
'The main point' is that you can't depend on the value of (EQ '(a b) '(a b)) being NIL.
Reason: a compiler compiles a Lisp file. The compiler sees literal data like '(a b) ' and '(a b). The compiler determines that these are 'similar' objects and 'coalesces' them into one object. It might even do it for '(a b) and '(1 a b), seeing that one is a sublist of the other and coalesces the first list into the sublist of the other.
Common Lisp compilers are explicitly allowed by the language standard to do these things with literal data in a source file. It defines a concept of 'similarity', where this is allowed.
$ sbcl
This is SBCL 1.5.8.138-d53fb0eb3, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
* (eq '(a b) '(a b))
NIL
*
/tmp % more test.lisp
(print (list :eq-or-not (eq '(a b) '(a b))))
/tmp % sbcl
This is SBCL 2.1.9, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (load (compile-file "test.lisp"))
; compiling file "/private/tmp/test.lisp" (written 03 JAN 2022 07:58:54 PM):
; processing (PRINT (LIST :EQ-OR-NOT ...))
; wrote /private/tmp/test.fasl
; compilation finished in 0:00:00.003
(:EQ-OR-NOT T)
TCCL and Lispworks return NIL for this test in both cases.
That is incorrect. It must always return false. LIST is called multiple times, and each time it is called it must produce a distinct cons. Note this is a different issue from literal coalescing (mentioned else-thread).
((ab . c) . d . nil)
isn't a valid s-expression. Maybe it should be
((ab . c) d . nil)It will give you the basic idea, but it won't tell you about macros, reader macros, compiler macros, and having the whole language always available to build your abstractions on; it won't tell you about CLOS, including multiple dispatch, metaclasses, or method combinations; it won't tell you about the condition system and programming from inside the debugger; it won't tell you about live recompilation and live coding; it won't tell you about the JVM interoperability of Clojure, Kawa, or ABCL; it won't tell you about Racket's tower-of-languages approach to programming; I can keep on going for a while. And, since most Lisp dialects and implementations are nowadays compiled, either to native code or to bytecode, the just-interpreting approach doesn't even work very well.
Articles like this are fun, but they end up completely missing the point. This isn't "the power of Lisp"; it's just the very, very, very beginning. The rabbit hole goes much deeper.
The best kind of comments are the ones which respond to the what as opposed to the who. :D
The Pascal one was used to build TeX, indeed.
Truly asking for help: can you help explain what can I do with macros that I cannot do with functions? Or, maybe, cannot do with high quality or low complexity using functions?
[1]
Some other languages support similar, if not more powerful, compile-time facilities (such as Lisp macros), but those are outside the scope of this article.
When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it.
In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon.
This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct.
[0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated.
[1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL.
Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears.
I'll keep learning. I'm sure the light bulb will go off eventually.
Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros.
Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure:
(when false (infinite-loop))
But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial.
(defmacro when
"Evaluates test. If logical true, evaluates body in an implicit do."
{:added "1.0"}
[test & body]
(list 'if test (cons 'do body)))
So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op.
This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it.
Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does.
Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages.
(defmacro with-open
"bindings => [name init ...]
Evaluates body in a try expression with names bound to the values
of the inits, and a finally clause that calls (.close name) on each
name in reverse order."
{:added "1.0"}
[bindings & body]
(assert-args
(vector? bindings) "a vector for its binding"
(even? (count bindings)) "an even number of forms in binding vector")
(cond
(= (count bindings) 0) `(do ~@body)
(symbol? (bindings 0)) `(let ~(subvec bindings 0 2)
(try
(with-open ~(subvec bindings 2) ~@body)
(finally
(. ~(bindings 0) close))))
:else (throw (IllegalArgumentException.
"with-open only allows Symbols in bindings"))))
By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it.
As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages.
I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try.
[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72....
[2] https://www.haskellforall.com/2012/12/the-continuation-monad...
Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros.
First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation
Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility.
Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp.
It's simultaneously very prosaic and very deep. Macros are pretty simple:
1. Access to the AST of a chunk of code for arbitrary processing.
2. At compile-time.
This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion.
Where Lisps are different:
1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures.
2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation.
3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally".
4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1]
These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build process for any compiled language. Eval, in interpreted languages, is as expressive as Lisp macros, but is incredibly poorly supported and unergonomic in comparison to Lisp macros.
It's an ergonomic and feasibility thing more than a capability thing. That said, POSIX shell is turing complete, so you don't need anything like C# or Lisp or Haskell or anything else we've discussed to get your computation done. The reaction you might have to comparing shell scripting to Haskell is not dissimilar to the reaction an experienced Lisp programmer might have to comparing AST manipulation and code gen through a library to Lisp's macro facilities.
[0]: Clojure, specifically abstracts the idea of callable, and there are more callable things than special operators, macros, and functions.
[1]: Languages such as Haskell and OCaml continue to push the boundaries of what can be handled by "standard function calls" (which term I hesitate to use, given its imprecision). That said, given that macros allow the use of a turing-complete language to generate inputs to the evaluator, they allow you to implement any language construct that is computable without modifying the evaluation infrastructure.
That's what drives me on. That and Rich Hickey's enthusiasm and the sense that he knows something really valuable that I don't. I watch a lot of videos as well, so I can't remember who said that programming languages were user interfaces between human and machine and are an attempt to talk to machines to get them to do things. In that sense, programming languages are a lot deeper than syntax convenience or efficiency. A good language allows us to talk to machine at a much higher level, and I get from functional programmers that this is what they find so attractive.
I will contact you if I get really stuck and when I finally get it just to share it with someone. Thanks!
edit: I've heard a lot about SICP but never looked into it. I think it's time, now that I have time
Videos: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
Text: https://github.com/sarabander/sicp (links to epub and html versions at the top of the README)
I don't have a computer science degree and I think that has held me back some, but I try to overcome it by always trying to learn and never thinking I should already know something or feeling like I'm an expert. I don't have the patience at my age to go back to school and get a degree. That you attempted SICP a few times is a good sign that you don't give up and speaks to your success in finally understanding.
Compared to Elixir (another language I like) - it is homoiconic but it doesn't have this feature. I can't, for example, take an arbitrary function definition and update or define it in the REPL - because all functions must be defined in a module. However, I like these modules because I think it helps enable discovery of functions - it's much easier to find all operations you can do on lists or maps in Elixir than Clojure.
Also, I wouldn't necessarily call macros more powerful than functions. They're just different, and they enable different things. You cannot, for example, pass a macro around like you can a function. Personally I've never written them, but I use them every day because the core library defines macros. I view them as ways to let people smarter than me to give me tools I can use to write my programs.
As far as driving you on, I don't think Clojure is the end-all be-all language. I happen to like it. I also like Elixir. I know people who really love other, more standard languages though. I don't think you need to force yourself to love or "get" Clojure, but playing around with it can expand your mind a bit, especially if you've never used an FP before.
But if you want an FP that (IMHO) would be easier to transition to, Erlang was the first FP that I "got", afterwhich I tried Clojure again. The syntax is a lot more like traditional imperative programs which can ease the transition. Nowadays I would recommend Elixir over Erlang for several reasons: more modern with modern tooling, less syntax quirks, more active online community, lots of active development.
For me, the main benefits of Clojure are (other FP languages have some of these too):
1. Sharable, immutable state as the default.
2. A set of higher order functions that can do all sorts of data transformation.
3. Being able to use the same language on the web in client + server (without having to use Javascript).
4. Clojure gives me the ability to, more than other languages I've used, target the level of abstraction that I need for the problem at hand, which removes a lot of cruft.
5. Lack of special syntax: helps enable REPL driven development, makes it easier to seamlessly adopt new features from other languages.
6. It's hosted on two ecosystems I'm very familiar with - Java and Javascript.
A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need.
But they are still normal functions - it's the time of execution that changes.
I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.
At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated.
There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false.
*See pg's _On Lisp_ for an in-depth discussion of macros.
What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)
That's why I stopped doing it, but I totally get that in low level programming that it comes in very handy in some circumstances.
As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary.
Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.
[0] https://coalton-lang.github.io/20211010-introducing-coalton/
1. Clojure does not have all of its various conditionals as language built-ins. There is a single conditional construct built in to the language, if. The rest are all defined in terms of if. Function arguments are eagerly evaluated in Clojure. How do you write a short-circuiting function in Clojure? (when predicate consequent) as a function must evaluate both the predicate and the consequent. If consequent is slow to calculate then your whole when expression is as slow as consequent, even if predicate is false and we do not need consequent. when is a Clojure macro based on the if special form.
2. core.async is a mere library.[0] This implements CSP (same idea as Go's channels) and async programming (the sort of stuff that needs compiler support in other languages such as C# or Go) as a library. If Rich Hickey didn't write core.async, you could build this control flow yourself for async programming. In fact, you can create arbitrary control flow mechanisms using macros. Common Lisp's entire object system can be implemented in macros.
3. The only special forms in Clojure are def, if, fn, let, loop, recur, do, new, ., throw, try, set!, quote, and var. The rest of what you would consider the core of Clojure consists of functions and macros. Much of the control flow functionality is implemented in macros.[1]
The core of macros is that they allow you to evaluate arbitrary user code at compile time. Clojure code is represented as native Clojure data structures (the same is true for all Lisps). The entire standard library which you are accustomed to using to manipulate standard data structures is available at compile time to transform the data structures that represent your code. I can't explain this better than Rich Hickey does in [1], so I encourage you to watch the linked section of that talk.
[0]: https://youtu.be/yJxFPoxqzWE?t=560 See this video for an overview of core.async. Later on he talks about the stuff that is implemented as macros in the library. This link starts up when he's talking about C# style async.
[1]: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get an overview of what macros are used for in Clojure's core.
(first-working
(get-it-from-the-cache)
(get-it-from-the-database)
(get-it-from-an-external-api)
(compute-it-the-slow-way)
default-value)
for f in GetFromCache, GetFromDatabase, GetFromAPI, Compute:
try:
return f()
except Exception as ex:
log(ex)
return default_value
def atdt_exception_suppression(*list_of_fns):
...
But macros are a powerful option because it doesn't matter if functions are first class or not. If you are using a language where functions aren't first class objects, a macro system would let Zak implement anyway.
So you are correctly identifying that Python can implement this in an alternate way as a function, but a macro system is sufficient to do this and a whole batch of other tricks even if first class functions aren't available as a feature.
def get_from_cache
get_from_cache!
rescue NoRecord
nil
end
def record
@record ||= get_from_cache || get_from_database || get_from_api || compute || default_value
end
Assume, for purposes of the exercise that someone else wrote some of the methods you're calling and they do not behave this way. You could write your own wrappers, but that's a lot more code to write and maintain than a simple control structure that doesn't care what's inside it.
With macros, I can write a short macro regardless of how the upstream code behaves. Without macros, I can request a change to the upstream code and hope the maintainer will agree to make it behave how I want.
It is also arguable that overuse of macros leads to less readable code since you wind up inventing your own personal language the more you do it, which requires more of a learning curve to work on it.
There's a large advantage to sticking with verbose idioms that are common and built out of a relatively few building blocks rather than building a meta-language which is terse.
At the same time I'm not saying don't have macros, but they should really be reserved for more extensive heavy lifting, not a pile of tiny ones used all over the codebase to make it terse.
A well written codebase should read more like Hemmingway at an 8th grade reading level using simple constructs from the base language which are put together, generally using programming design idioms which are common and terminology which everyone shares. There shouldn't be a lot of "what does this macro do and where it is it defined?" questions on every other line.
In Common Lisp:
(defmacro return-first-working (&body body)
"Expects a sequence of expressions as body.
Returns the value of the first expression of body
which returns without error."
(when body
`(handler-case ,(first body)
(error ()
(return-first-working ,@(rest body))))))
(return-first-working
(error "hi")
(error "there")
(error "!")
"hello-world"
(error "wtf?"))
Above is a recursive macro, which takes its subexpressions and returns a simplified expression, reusing itself. That way code transformation itself is described as a recursive process. One can learn how to apply such transformation patterns to the code itself and then it gets relatively easy to write.
(first-working
(get-it-from-the-cache (or local-cache default-cache))
(get-it-from-the-database current-database-connection)
(get-it-from-an-external-api (ask-user-for-api-credentials))
(compute-it-the-slow-way foo bar baz)
default-value)
There's also an advantage in clarity because an abstraction can be defined. Once you know what first-working is for, the purpose of this block of code is clear as soon as you read the first token. Using a pattern instead of an abstraction, it's necessary to read the whole loop to make sure it's really the pattern you think it is.
firstWorking() ?? getItFromTheCache() ?? getItFromTheDataBase() ?? getItFromAnExternalApi() ?? computeItTheSlowWay() ?? defaultValue;
Perl OR short-circuit:
firstWorking() or getItFromTheCache() or getItFromTheDataBase() or getItFromAnExternalApi() or computeItTheSlowWay() or defaultValue;
cache.get
.orElse(db.get)
.orElse(api.get)
... const firstSuccessfulOptionValue = fnReturnsOption
.orElse(fn2ReturnsOption())
.orElse(fn3ReturnsOption())
.orElse(fn4ReturnsOption())
.getOrElse("val");
Spend 5 minutes playing with -> [0]. Appreciate how pretty it makes the code and how nice it is for laying out sequential operations. What a fun operator. Then try to implement it yourself as a function.
When you discover that you can't, you will have discovered the unique power of macros (note the docs link to source code so you can see the readable 8-line macro implementation). You've also learned a very practical skill, which is how to write threading macros that handle your special edge case fail conditions. It comes in handy from time to time.
[0] https://clojure.github.io/clojure/clojure.core-api.html#cloj...
But I don't think this is a bad example at all, I tend to think of macros as "extending the language/compiler", since they have access to the abstract syntax tree of their inputs, which is what is being done here.
It doesn't matter what specific combination of language features are supported. Macros mean you can implement things anyway. It doesn't really matter what the evaluation logic of the language is, a macro can create syntax that does what you want it to.
If there is a macro system available, a new operator can be implemented in any language. If there is no macro system, it depends what features the language supports.
One fairly simple example is Clojure's `when`, which is an expression that starts off looking like
(when (= 0 1) (+ 1 2))
(if (= 0 1) (do (+ 1 2)))
It's instructive to see how much of e.g., Clojure is implemented as macros. https://github.com/clojure/clojure/blob/master/src/clj/cloju... shows pretty clearly the correspondence between the original form and the rewritten form in the definition of `when`. `defn`, which is so foundational to idiomatic Clojure that it's easy to mistake it for a built-in, is a much more complicated example.
(or a b)
Macros can analyse/manipulate the abstract syntax tree of their inputs, so I think of them as extending the compiler.
Now you only need to define the function OR...
It's possible to write pretty complex things like control flow abstractions using just structures and functions/methods, e.g. a CL-esque condition system in Java which only uses classes, static methods, and Java lambdas[0] for its syntax. Possible, but also IMO ugly when compared to the CL counterpart, because the low-level but irrelevant details (such as instantiation via `new` or generics) are still presented to the programmer.
I plan to fall down the LISP rabbit hole one day, and continue this series in full detail.
Maybe this is true for language hobbyists or whatever, but every few years throughout my career, I went back to LISP or LISP-like languages and never found it super profound. Do I need to spend more time on a project? I just don't get the same excitement others seem to get.
It depends on your interest. For me in particular Lisp was a "profound enlightenment experience" because:
1. I have been programming for a long time and had so many problems I could not(in practice) solve in any language but Lisp.
2. Lisp gives you two things no other language gives you: very basic Metaprograming(it can read itself) and the ability to design your own DSL(domain specific language).
3. It also gives you REPL(Read eval print loop) immediate feedback other languages tried to copy(but have not been able to replicate entirely).
Having said that, Lisp design is also obsolete today because it was mainly designed in the 1970s. I hate it (specially parentheses and interface) and love it at the same time.
>Do I need to spend more time on a project?
You don't need to do anything, nobody needs to learn a new language for living.
The fastest way to understand what Lisp is all about is to be near a very proficient Lisp programmer, because he will be able to run circles around you solving problems.
That combination of someone proving to you what is possible at the masters level and natural human competition is probably much more powerful than anything else you could do on your own.
We are humans after all, we see a great hunter hunt and only then we desire to be as good as he is and learn from him. We do it subconsciously and that is the reason elite tennis players sell razors so well, people copy naturally those who they admire without being conscious about it.
Other stuff is that it tends to take some time working, and actually ending up using some of the more interesting bits, to see effect.
Lots of people for example talk about macros when it comes to homoiconicity, but rarely (outside of old lisp wizards) do I see people going further to how Lisp makes it easy to build languages and compilers that are easy to debug and handle - I'm not just talking about the common idea trope of "write DSL for your program", because a) you're going to write a partial language specific to your program anyway, no matter the language, you might just obfuscate it calling them functions and types b) the easy manipulation of code and lack of parsing (and thus poof goes half the dragon book) means that writing compilers is simply... simple. And with CL's macroexpand and similar features in other lisps, it tends to be much easier to figure what is going wrong (and it's reasonably easy to add reporting of errors in compiled code down to specific form)
Still, the basic eval command looks remarkably similar in lisp vs TCL.
Can you elaborate? I am not sure what you mean.
It sucks. Tcl will break you down.
Not original commenter, btw.
Compared to that, TCL's composite type is "list", which has an arbitrary number of elements, and recursive iteration is not as simple. Yes, one can use "lindex"/"lrange" to do LISP-style recursive iteration, but this is not idiomatic at all and has horrible performance. Instead, people use foreach/explicit numeric indices, and this needs more support and special iterating-related functions, see [0] for example.
[0] https://wiki.tcl-lang.org/page/Map+in+functional+programming
The primary difference is whether arguments are evaluated before (Scheme/Lisp) or after (Tcl) being sent to the function/command.
The deferred evaluation is what allows you to do things that you need macros for in other languages. (ie. you need deferred evaluation or macros to implement short-circuit operators, for example)
However, early Lisps had FEXPRs which also deferred evaluation. The late John Shutt built and entire Lisp called "Kernel" around the idea:
http://web.cs.wpi.edu/~jshutt/kernel.html
I strongly recommend a read through his PhD thesis: https://www.wpi.edu/Pubs/ETD/Available/etd-090110-124904/
Unfortunately, his implementation isn't anywhere near as good as his conception. Other people have implemented it better.
http://axisofeval.blogspot.com/search/label/kernel http://axisofeval.blogspot.com/2011/09/kernel-underground.ht...
The Oh Unix shell grabs some of the core Kernel concepts and runs with them in a useful package: https://github.com/michaelmacinnis/oh
In the lisp terms, imagine a language with no macros and with immediate evaluation ("applicatives" in the Kernel-speak). And no special forms either, except for syntax-level quote command. This means that, for example, for "if" you have to quote the code parameters to make sure they are not evaluated too early:
(if var '(print '(hello world)) nil)
And that's how you build a language with homoiconicity and macro-like facilities, but without FEXPRs or applicatives or special syntax. Granted, it has plenty of downside, but it's still a very interesting approach.
1) Arguments are not automatically evaluated before being passed to the underlying "operative" in Kernel-speak. This is important because it obviates the need for the macro machinery. This is fairly similar to Tcl.
2) Environments are first-class in Kernel. This is important because you can now constrain the scope explicitly. Do I want to evaluate this in the defining context (lexical scope)? Do I want to evaluate this in the calling context (dynamic scope)? Do I want to evaluate this is global or initial scope? All of these things are possible because environments are a thing that can be passed around.
The downside is that this level of mutability seems to constrain the ability to pre-compile things.
One of the "problems" that I would consider Tcl to have is that certain "atoms" don't self-evaluate. There's really no good reason for why you have to write:
% if 1 {expr 2} else {expr 3}
instead of
% if 1 2 else 3
And no good reason why you can't write:
% + 1 2
3
instead of
% expr 1 + 2
3
This seems like going gratuitously out of the way to avoid "looking like Lisp" which was in its down phase by 1990.
I feel most examples just are the same list-manipulations / Fibonacci- / prime number calculation ad nauseam.
Are there any active projects out there that Lisp family programmers would recommend as a good example of what the language can do and be used for (preferably with common elements such as networking, db connections etc)?
IntelliJ has nothing close to it, does it?
Nope, saw nothing for IntelliJ. However if it highlights parens, one can code in there and `(load "myfile.lisp")` in any other REPL. It's a start.
The only thing you would need to give up to drop them altogether is &key/opt/rest args.
I've been a serious Lisp programmer (recreationally and sometimes professionally) for around 10 years and I have programmed in Common Lisp, Scheme, my own weird dialects, Emacs Lisp, etc.
At this point I feel like almost everything written about Lisp is silly. Taken as a language family as a whole, there isn't much that separates Lisp from most of the other languages that are out there _except_ syntax-transformations or macros which, frankly, most people should never use.
For me, the best thing about Lisp is the regularity of the syntax. What I like least about almost all other programming languages is all the useless syntactic doodads and wingdings. The brackets, the indentation sensitive stuff, the unneeded complexity that languages insist on associating with what is and must be the denotation of a tree, in the end.
The other great thing about the language is that you must use `let` or `lambda` to introduce variable bindings and their scope is, therefore, always perfectly clear.
Despite these niceties, Lisp isn't going to make you into a super programmer. It won't really help you solve hard problems. It might help you quickly solve easy problems, but in the end, most of the job of a software engineer is grappling with conceptual problems with which Lisp is only sort of helpful.
As long as I'm holding forth, I will say that I enjoy programming in Scheme _much_ more than I enjoy Common Lisp, which feels incredibly, uselessly, and old-fashionedly complex with all sorts of weird corners to stub your toes on.
Many macros—just like programs as a whole—are bad and unhelpful. But this has little to do with macros, and more to do with programmer technique. It's not intrinsically difficult to write a useful macro, but it does presuppose that the programmer has a useful syntactic idea.
The Lisp family of languages so so broad and loose. It's hard to say anything generally about them anymore. But if you narrow down to Common Lisp, there's still a treasure trove of unique features. Common Lisp and Clojure seem to be the only two lisps where you've achieved broad community agreement on tooling, ecosystem, and general library design. They're also two languages where you can get paid to write them. Can't say the same about Scheme.
Lastly, as a personal anecdote, I don't think I've stubbed my toe on anything in Common Lisp. It's a very safe language with a lot of language features. I have stubbed my toe with CALL/CC though and memory leaks.
I think you're right about CL and CLojure, though. While I really prefer Scheme, its not really appropriate for solving a large set of practical problems.
I have to take issue with this. There are only two languages I consider when faced with a very hard problem: Common Lisp and Haskell. One can solve easy problems in any language. But when I have to solve a new problem from scratch that nobody's ever solved before, I don't reach for Java or Python. I reach for the languages that augment my brain rather than limiting it.
I want a language that has one and only one obvious way to do things. That doesn't require building my own language features or choosing an addon that might or might not be compatible with others.
I don't have any use for the flexibility to customize a language because I want a language that doesn't need to be customized.
If I was doing cutting edge AI i might feel differently, but I'm not. The stuff I do has only one place for creativity and that's the UI. The code should be boring and utterly forgettable. If someone remembers your code, it's probably because they had to spend time looking at it when they could have been writing unit tests or moving on to new features.
My key take aways from learning how LISP is built: 1) Code & data can be unified into the same thing. This isn't particularly deep if you know the concept of an AST. 2) You can bootstrap a language from a simplified version of the language. Repeat this a few times and you get a fancy language. This isn't particularly deep if you've built any compiler, or if you've built LaTeX. 3) You can build beautiful things if you disregard the grittiness of reality. LISP offers an abstraction of physical computers that does not at all resemble a real computer. Some people like this abstraction. Maybe because they don't like how computers actually work.
You clearly don't, since your aim is to write boring and forgettable code. I don't think there is anything wrong with that -- you can write boring code to solve interesting and worthwhile problems, and overall the tendency is probably to err on the side of getting distracted by hip or shiny tech. But some people enjoy writing interesting code, and sometimes interesting code is needed to solve interesting problems and a powerful language can make certain types of interesting code much less painful to write. Auto-differentiation might be a good example.
It's mostly way above my level though, so for now I'll stick to solving the trivial stuff, so the real geniuses can get back to work.
Nobody might ever say "Wow that's some great code", in fact they'll probably say it's bloated and mediocre... but they'll be glad I thought to put an override button for that one edge case of the super trivial thing they thought was fine!