lispm

> I think one of the reasons recursion is often emphasized in relation to Lisp is because one of Lisp's core data structures, the linked list, can be defined inductively

Lisp was used in computer science education to teach "recursion". We are not talking about software engineering, but learning new ways to think about programming. That can be seen in SICP, which is not a Lisp/Scheme text, but a computer science education book, teaching students ways to think, from the basics upwards.

Personally I would not use recursion in programs everywhaere, unless the recursive solution is somewhat easier to think about. Typically I would use a higher order function or some extended loop construct.

It's also not that relevant. You'll can add different syntax variants, the Lisp evaluator doesn't see parentheses in any way, just actual interned data.

The Lisp compiler will need to figure it out and complain about various problems: undefined function/variable/tag/..., argument list mismatch, type mismatch, etc.

The compiler will expand the macros at compile time and the generated source code then is checked.

One thing this macro system enables are macros which can implement relatively arbitrary syntactical extensions, not restricted by a particular syntax. That can be seen as useful&flexibility or as potential problem (-> needs knowledge and discipline while implementing macros, with the goal of ensuring the maintainability of the code).

An "AST in spirit" is in our mind, but not the machine.

    (let ((s (s s)))
     (flet ((s (s) (declare (type s s)) s))
      (tagbody (s) s (go s))))

The function READ has no idea what S is: variable, operator name, data symbol, type, go tag, function name, macro name, ...? For each above we don't know what s is. We would need to parse it according to some syntax (and possibly know the current state of the runtime.

An AST would be the product of a parser and the parse would parse the code above according to some provided syntax. The AST then would encode a tree built from some syntax, where the nodes would be classified. In Lisp symbols and lists have multiple uses and the s-expression does not encode which uses it is: is it data, is it some kind of syntactical element. There is also no defined parser and no syntax it encodes on that level. READ is at best an s-expression reader, where it has zero knowledge what the lists and symbols supposed to mean in Lisp. For the reader (+ a b), (a + b), (a b +) are just s-expressions, not Lisp code.

It could theoretically be a graph.

For example I can make a circular list being a part of the source and the macro may process it.

Get the first two items from a list and add them:

    CL-USER 21 > (defmacro add-2 (a)
                   (list '+ (first a) (second a)))

ADD-2

Now we construct a circular list and use the macro:

    CL-USER 22 > (add-2 #1=(1 2 3 . #1#))
    3

The circular list is really a part of the source code:

    CL-USER 23 > '(add-2 #1=(1 2 3 . #1#))
    (ADD-2 #1=(1 2 3 . #1#))

Another example: One could pass in a string and have the macro parse the string...

> Lisp ... macros typically operate on AST

In Lisp they don't. See Emacs Lisp, Common Lisp, ISLISP. A Lisp macro gets only passed some data and returns some data. There is nothing like an AST.

If we define a macro foo-macro and we call it:

    (foo-macro ...)

then ... can be any data.

Example:

    (defmacro rev (&rest items)
      (reverse items))

Above macro REV gets data passed and reverses it. The &rest list of ITEMS gets reversed. ITEMS is the list of source arguments in the macro call.

We can then write:

    (rev 1 2 3 4 +)

    (rev (rev 10 n -) (+ a 20 b) (rev 30 a *) list)

Example:

    CL-USER 7 > (let ((n 4) (a 3) (b 2))
                  (rev (rev 10 n -)
                       (+ a 20 b)
                       (rev 30 a *)
                       list))
    (90 25 -6)

There is no syntax tree. All the macro does, is to reverse the list of its source arguments.

If I let the macro describe the thing it gets passed, then we get this:

    CL-USER 8 > (defmacro rev (&rest items)
                  (DESCRIBE items)
                  (reverse items))
    REV

    CL-USER 9 > (rev 1 2 3 4 +)

    (1 2 3 4 +) is a LIST
    0      1
    1      2
    2      3
    3      4
    4      +
    10

As a side effect, we can see that the macro gets a simple list passed. A list of numbers and a symbol ("symbol" is also a data type). Not text. Not an AST.

Also data which is possibly not read, but computed by other code. We can compute the arguments of the macro REV and pass the thing to EVAL. Works the same as above.

    CL-USER 10 > (eval (append '(rev) (loop for i from 1 to 4 collect i) '(+)))

    (1 2 3 4 +) is a LIST
    0      1
    1      2
    2      3
    3      4
    4      +
    10

The macro only gets that data and can do anything with it. There is no idea of Abstract Syntax Tree: it does not need to be a tree, it does not need to be valid Lisp code and it carries no syntactical information. Generally, it also does not need to return valid Lisp code. To be computed all we need is that the evaluator eventually sees valid Lisp code, not the individual macro.

In Lisp the "reader" by default only parses a data layer: symbolic expressions. EVAL, macros and other Lisp functionality gets mostly data passed. EVAL has to figure out, what (a b c) actually is as a program. It can traverse it in an interpreter or compile it. A compiler may internally create AST representations -> but that free to the implementation.

The Lisp language then typically is not defined over text syntax, but data syntax.

A Lisp interpreter processes during execution not text, but s-expressions. It's a "List Processor", not a Text Processor. Lisp not Texp. ;-) The function COMPILE gets an s-expression passed, not text.

Racket and Scheme have other macro systems.

The precursor to CLIPS was called ART Inference and was a large commercial & very expensive Expert System development tool, written in Common Lisp. Unfortunately it seems to be lost - I haven't seen anything about it for several decades.

Other than that there are a bunch of rule-based systems in Common Lisp.

For example LispWorks Enterprise includes "KnowledgeWorks", which features a forward chainer based on the RETE algorithm.

https://www.lispworks.com/products/knowledgeworks.html

> Everybody implementing their abstractions, every library implementing their own language

In Lisp one can extend the language syntax. One does not need to implement a new language. For example the Common Lisp Object System adds operators like DEFCLASS, DEFMETHOD, DEFGENERIC, ... in addition to already existing operators like DEFUN, DEFSTRUCT, DEFTYPE, DEFPARAMETER, ...

Thus the language can be incrementally extended, instead of using a completely new language.

The CLOS macros are used by many programmers, they were standardized, documented and implemented.

As every form of abstraction, syntactic abstraction needs to be learned and requires extra work. Lisp was from the start developed to enable such things, and its early form of that applied to Lisp itself were the Lisp interpreter evaluating s-expressions, FEXPRs (procedured receiving unevaluated arguments) and in 1962 macros.

> I feel like Lisp and Haskell attract people that love programming more than shipping code.

One of the purposes of Lisp was to implement new ideas: like computing with mathematical formulas (-> Macsyma and earlier attempts), computing theorems (-> ACL2 and earlier attempts), ... This ability to support experimentation in R&D (actors, rule-based systems, frame systems, ...) was always a part of the language community.

If we look at the surviving commercial systems (Allegro CL and LispWorks), they still support this feature set.

Clojure was developed as a Lisp-inspired functional language with deep Java/JVM integration to support enterprise programming for people who were tired of programming in Java. That's fine. But the advice then was to not utilize the full power of Lisp (syntactic abstractions), to address the fear of unmaintainable software, while at the same time not using various features of Lisp (like gradually typed implementations of Lisp like SBCL) which helps to write robust software.

Generally Lisp has a broader outlook. It ran on an experimental spacecraft, it powered autonomous robots inspecting pipelines, it was the base for early research in computing (-> Interlisp), it ran on calculators (HP RPL), it supported children education (Logo), it's used to schedule telescope operations (-> Hubble and James Webb), it is used to schedule airline/airport operations, ...

> php has more killer software than all those languages combined.

If we research for ground breaking software, then I bet over the long history of Lisp there were a lot more in many more diverse domains written in Lisp

Lisp has been applied to develop airplane parts. BOEING and Airbus used to be Lisp users (they might be still use it for older planes). ICAD was the first parametric CAD system. ICAD used macros to describe physical objects and their relationships. This was used to construct turbines, wings, and a lot of other parts of Airplanes.

If you look into the long history of Lisp, it has been successfully applied in domains PHP has never seen. Sometimes really crazy stuff, like the first virtual military troop training systems, where the US army deployed a few extensive training and simulation systems, where the virtual worlds were controlled & generated by a Lisp system and rendered by multiple graphics engines for each training simulator. In the early 80s. -> https://en.wikipedia.org/wiki/SIMNET

That's nothing like PHP, but it was ground breaking in its domain. PHP had a different purpose: it was developed to make it simple to write&generate web pages in a large scale. That's a huge domain, but that was not what Lisp was developed for.

Btw., we see a bunch of new languages which now also support some form of macros. A recent example is Rust.

And the possibility of using "Google" to check for name collisions seems not to be known...

"toxic" is also to generalize from one person / one forum, to an extremely diverse group, many which never used Usenet

> major weakness is that it makes everythign unergonomic in order to make macros very ergonomic.

Generally Lisp has two main goals:

* allow code to be data and data to be code

* make coding and coding code (let code manipulate code) interactive

The s-expression based syntax was found to be useful for both. For the latter it means that code can be manipulated interactively for example by structure editors or in read-eval-print-loops which work with data. That makes interactive code writing very ergonomic.

The purpose of a vector in Lisp is to provide contiguous storage with integer index addressing and O(1) access time.

Cons cells are typically used to provide lists with O(n) access time.

compare the last release dates

    2012 Allegro CL  9.0
    2015 Allegro CL 10.0
    2017 Allegro CL 10.1
    2024 Allegro CL 11.0

    2012  LispWorks 6.1
     2012  LispWorks 6.1.1
    2015  LispWorks 7.0
    2017  LispWorks 7.1
     2018  LispWorks 7.1.1
     2019  LispWorks 7.1.2
    2021  Lispworks 8.0
     2022  LispWorks 8.0.1
 

Looks similar to me. LispWorks also had patch releases in between. Both are on the market for more than 35 years. Both are mostly written with CLOS and thus are especially to update with patches, additionally to the usual ways to update code (-> late binding). One just loads patches (which are mostly compiled Lisp code) into a running Lisp and that's it. Alternatively one can save a new image with patches loaded.

When one needs a patch or a feature, one would typically contact them directly. Both provide patches to the users.

Franz has made that simple for the user, they have a relatively continuously stream of patches, one can call an update function and it gets the necessary patches and installs them.

SBCL has monthly (!) releases, where the user (that's what I do) would typically compile it from scratch using the supplied sources. Updating is quick, around a minute for a recompile.

LispWorks still exists and works great.

cons cells are not the base for vectors, strings and a bunch of other datastructures. Cons cells are mostly only used for nested lists or similar.

IMHO, the Lisp Machine experience is completely different. Those have zero educational aspects, they were designed to write large, and stateful, applications, where the Lisp Machine OS is the big stateful object-oriented operating system (like Symbolics Genera). Interlisp-D is even more different.

I totally agree.

Maybe it's not "technical issues", but features and support? Allegro CL has a proven GUI toolkit, for example, and now they moved it into the web browser.

FYI: here are the release notes of the recently release Allegro CL 11.0: https://franz.com/support/documentation/current/release-note...

IIRC, Cyc gets delivered on other platforms&languages (C, JVM, ... ?). Would be interesting to know what they use for deployment/delivery.

The standard hasn't been updated. But progress happens in implementations and libraries.

Which makes it very different in Racket. STRING-REST creates a new string with the contents copied from the original string, minus one character.

Part of your expertise is not the language, but the application to embedded platforms.

Looks problematic. Is a string in Haskell a list? I would think that in Racket a string is a vector. FIRST and REST operations then would have very different implementations.

References:

Lucid Inc. used the word internally at least since 1987, mentioning a treeshaker for Lucid CL 3.0 on the VAX.

"Lisp Systems in the 1990s", Layer/Richardson, 1991 https://dl.acm.org/doi/10.1145/114669.114674

From above: "In contrast, tree shaking uses the approach of eliminating --shaking out-- what is not needed in a static fashion. Programmers specify the requirements of their application and the unneeded parts of the Lisp system are removed using their detailed knowledge of the application. The disadvantages of tree shaking are that it requires programmer intervention and that a function which is shaken out cannot be easily restored."

"Building Common Lisp Applications with Reasonable Performance", Boreczky/Rowe, 1993, https://dl.acm.org/doi/10.1145/1040032.174174

"Lisp: Good News, Bad News, How to Win Big", Gabriel, (not sure from when this version is, the original article is from 1989) https://www.dreamsongs.com/Files/LispGoodNewsBadNews.pdf

From Gabriel's essay:

> "1.6.3 The Treeshaker

> Most Lisp development systems, including Lucid’s, provide all the resources of the Lisp system by default, and this in turn leads to a style of development in which the programmer makes use of whatever tool happens to be most convenient. Because much of the basic Lisp system (or any development system built on top of the basic Lisp system) will generally be unused by a given application, it is very worthwhile to have a tool for excising these unused parts. This tool is called the Treeshaker.

> Treeshaker execution occurs in three phases: walking, testing and writing. In the walking phase, the Treeshaker accumulates a set of objects that need to be included in the saved image. After making this set, the treeshaker runs a test of the application to check that all objects which are used in a typical run have been included. The writing phase then generates an executable image which will run the application.

> To a first approximation, the walk phase is just a matter of computing the connected component of the Lisp image (treated as a directed graph in the obvious way) generated by the application’s toplevel function. However, because of the way that Lisp objects are generally connected this usually includes almost the entire Lisp image including the unused subsystems. Therefore the treeshaker uses several techniques to find connections between objects that do not actually need to be followed in the walk."

> "The name Treeshaker is meant to be evocative of the idea of actually shaking a tree to dislodge dead branches or other trash."

On the more funny side, LispWorks has a keyword to the DELIVER function :shake-shake-shake , which invokes the treeshaker during application delivery.

https://www.lispworks.com/documentation/lw80/deliv/deliv-key...

One is pruning dead stuff and shaking the tree is then letting them fall down. The garbage collector then takes it away.

The name "treeshaker" for an application delivery tool possibly originated in Lisp. The first time I've found it was in Lucid Common Lisp, a (no longer available) commercial implementation of Common Lisp for UNIX. Lucid CL 4.1 in 1992 included a tool called Treeshaker. Lucid CL was one of those Lisps which have the idea of an image (see for example the "image" feature of Lisp 1 in 1960), a saved memory dump of the current heap of a running Lisp. An application then is an image plus a runtime. The image typically includes almost all code AND data from the memory, which sometimes creates the wish to create smaller images for application delivery. Lucid CL then included a tool, which before saving that image, removes all kinds of unused code and data, depending on some meaning of what „unused“ means. For example the symbol table may have variables, types, functions, etc. which are not referenced anywhere. A treeshaker tool might also get a list of things to remove. Thus in a graph of reachable Lisp data&code, the connections are pruned. Either a GC or a specialized piece of code then collects the garbage, shrinks memory (-> shakes the tree and everything which is cut loose is falling down) and dumps it as a possibly smaller image.

The treeshaker thus was not a compiler tool, but a tool to remove what was determined to be unused code of a Lisp heap. Remember, by default such a Lisp image would also contain a compiler, an interpreter and an implementation of a read-eval-print loop. Thus if we break (-> interrupt a thread which then provides a REPL) a running program into such a read-eval-print loop, we could still use all the code, which is in this heap (which was restored from an image). Thus it would make sense to remove the compiler too, and possibly the read-eval-print loop, too.

I would just use an existing Common Lisp library, which provides query and select.

> voters can directly influence political decisions

and the "elite" can influence voters through other means. That "masses" can be manipulated should be known.

> the will of the people

women were not "people"? Were they asked? If not, why not?

> it's the majority of the real population

I have questions. What is a "real" population?

The swiss direct democracy is about "the majority of the real population" ? So the intiatives are won with a majority of the population? >50% of the population? Is that true? I thought it's something like >50% of the valid votes.

> You should inform yourself about this great political system.

Please explain this great political system to me.

wonderful!