Modern, functional Common Lisp: myths and best practices | Hacker News Reader

Modern, functional Common Lisp: myths and best practices

ambrevar.xyz

226 pointsAmbrevar155 comments

Common Lisp is also a pleasure to use for k8s-hosted services. Just deploy your Lisp-based service to k8s, forward the sly/slime port to your local system, and continue to work on it interactively with emacs and sly/slime. It's really the most interactive development approach for k8s-hosted services. See https://github.com/container-lisp/s2i-lisp

How do you do the forwarding? (That’s an interesting question to me by itself, but I ask because it seems like that feature is just as good outside of k8s too :))

With k8s, you just create a service of type NodePort and it assigns a free external port for that service, which is mapped back to a specific port in the Lisp container.

I know of NodePort, but last i checked it’s a low-level constrict that doesn’t do anything like, say, authn—so that doesn’t seem like a complete answer.

What sort of encryption and authentication can I expect if I do that? Can anybody who guesses the port dump code into my running Lisp instance?

This is entirely to hard to find the answer to in the kubernetes/kubectl documentation. Apparently the forward is done via socat over tls:

https://stackoverflow.com/questions/50645059/is-kubectl-port...

And socat is mentioned in:

https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/po...

I didn't actually manage to find out what or how socat is called - but I'm also on mobile, so it's a little convoluted to browse source code.

Anyone know where the call to socat happens, after kubectl port-forward?

The encryption and authentication that the Lisp image exposes :)

Someone else mentioned that kubectl port-forward also adds these for you, so if you are not in a private cluster, it may be a better option.

Don't do this... You only need the connection for development purposes, so use "kubectl port-forward", which gives you the auth you'll need as well.

Outside of k8s, which I'm not familiar with, ssh can do port forwarding.

Sure, I know general techniques and k8s-specific ones (and have used production SSH-to-programming environment REPLs with twisted). I’m asking partially because my job these days is to secure setups like these :)

kubectl port-forward my-lisp-pod 4005:4005

then M-x slime-connect localhost 4005

ivan4th had the right answer. Use the "kubectl port-forward" command.

BTW, here's a k8s-hosted app I wrote using this approach: https://github.com/atgreen/red-light-green-light

Generally, here's how to connect to a remote running lisp image, and send changes from one's comfortable editor: https://lispcookbook.github.io/cl-cookbook/debugging.html#re...

Thanks for writing that up. Common Lisp as a language and ecosystem is so huge that we all have our own view of what CL is and how to use it to build applications. I have been actively using CL since around 1982 for prototyping and also building applications and tools but I feel like I use a small part of what is available, mostly because I prefer to use what I am used to. I should should probably spend a little less time building things with CL and a little more time studying it.

EDIT: off topic, sorry, but I have been actively evaluating CL (using an embedded web server that starts a browser) vs. Swift (using mostly SwiftUI) for a new product I want to write. I find myself using Swift like I would CL: using Playgrounds to prototype low level code and utilities, then XCode for developing the UI. To be honest, Swift and SwiftUI is a better fit technology-wise for what I want to do, but I am so much happier when working in CL.

Can you elaborate why is it a better fit?

Have you evaluated CAPI? Have you looked into building Cocoa applications with Clozure CL?

CAPI is great, very well designed and implemented.

The advantage of Swift and SwiftUI is that my application can run on both a MacBook and iPad, sync data with iCloud.

The advantage of LispWorks is that I could fairly easily support both Mac and Windows (and Linux).

I spent two evenings last week playing with Clozure CL and its Cocoa support. It has low level APIs and I had a peculiar issue with sometimes not being enter text into input components.

EDIT: to be clear, in general I love Clozure CL. In this instance I was probably shooting myself in the foot, somehow, or maybe the issue was that I blew the install in some way getting it running on Catalina which took over an hour. The Catalina problems are hopefully just short term issues.

I haven’t used it myself, but EQL5 is attempting to make it possible to use ECL to iOS and Android apps

LispWorks also has offers an iOS/Android runtime, but no CAPI unfortunately.

Yeah, EQL5 can be used for UIs, if I understand correctly because it builds on ECL to compile CL to C

The iOS runtime product allows you to write native libraries, but not do UI directly.

CAPI supports at least Mac, Windows, and Linux.

BTW, it just took me about 40 minutes using CAPI to reproduce what took a couple of hours this morning doing the same UI in Swift and SwiftUI.

> Common Lisp as a language and ecosystem is so huge

This is not true IMO, when compared with the languages of today. Even if you include the numerous little details of MOP, the CL spec is nowhere near as confusing as say something like C++ 17.

You are correct, of course. Java, Python, C++, etc. also have huge ecosystems.

I've been doing some functional style programming in Common Lisp, and I was wondering what exactly should be considered functional programming.

In particular, is object identity with EQ consistent with functional programming? Constructors do not act like functions if EQ is the equality. Or should that be more "immutable programming"?

Common Lisp, because it has EQ and object identity, cannot perform some optimizations that a truly functional language's implementation could. In particular, it cannot combine equivalent function calls, and cannot merge equivalent data (hash consing or the equivalent.)

Immutability is something I'm still exploring in Common Lisp. Any pointer, anyone?

It's a matter of not modifying things once constructed. CL has hooks to help do that, although a user can always get around them.

A package I've been involved with lately is fset, which is available through quicklisp, or at

https://github.com/slburson/fset

It has some interesting features, including "functional setf expansion". This would turn something like

(setf (fcar x) y)

into something equivalent to

(setf x (cons y (fcdr x)))

(where "fcar" and "fcdr" are the same as "car" and "cdr", except when they are in a setf-able place form.)

(fcar and fcdr are not in fset; I used those names just for exposition here.)

It could work with nested accessors, but only if it bottoms out in a variable.

Are you aware of any memory-leak issues with fset? I once used it in a game where I updated an fset sequence in real-time. I let it run for a while, after coming back I noticed my computer was completely frozen (out of RAM).

It's inspired by Clojure after all. The language of choice for users that don't care about performance.

It probably isn't. The initial public release of Clojure was in the fall of 2007. Scott Burson has been working on FSet since at least 2004. He himself says it's inspired by Refine.

I stand corrected. I still maintain my point about (idiomatic) Clojure being terribly slow though.

And terribly beautiful ;)

Just personal preference and enjoy programming in it. Programmers give too much importance to programming languages.

Clojure can be made to run very fast (it compiles to the same bytecode as Java after all) for the hot paths at the cost at writing less idiomatic Clojure.

Didn't know about this, it looks very cool indeed! Thanks for sharing!

https://common-lisp.net/project/fset/

By the way, I found a lot of your best practices questionable to say the least. Since it's obvious that you are a newcomer to CL, I would refrain from producing "best practices" type blog posts until I had a few years of experience under my belt.

For that reason, I also found your post confusing and I'm inclined to categorize it as "mostly rehashing stuff that is already there" rather than strong signal. There is ample, good, introductory material for CL on the net, we should strive to think before we dilute it with derivative posts.

Thanks for the fset link.

I understand your comment as a bit judgmental, if not rude. Please keep the tone friendly, it helps if we want to have constructive discussions. I'd be happy (as well as everyone around here I'm sure) to discuss what you find questionable.

I agree with GP about refraining from "best practices" type advice -- at most leave it as "I prefer doing it this way". (Example: There was no mention of qlot for a form of version pinning, which would be considered a "best" or at least "standard" practice in other ecosystems, but while I use it I'm not going to insist other Lispers I don't work with do so too. They can keep doing things how they like. Common Lisp is not for the strongly opinionated.)

Apart from that though I do think it's useful to have the occasional new posts showcasing rebuttals to a few "myths" that won't die, so I appreciated your post. Some gifs of editor interactivity (like in https://malisper.me/debugging-lisp-part-1-recompilation/) would make it better. Though ironically (one myth has been "it has no libraries") perhaps some people might be turned off by the mention of so many libraries ("ok such-and-such is there after all but I have to type a quickload, why can't it come without any work?" -- programmers are often lazy in the worst ways). Before using every utility library under the sun and then some, I'd encourage people to get to know the base language... It's typically quite sufficient for many tasks. But hey, libraries are cool too.

Two other persistent data structure libs that came up in a thread the other day (though fset seems to be the most popular): https://github.com/danshapero/cl-hamt/ and https://github.com/ndantam/sycamore

Right, maybe "best practices" was poorly worded. I'll think of something better.

Regarding the visual aspect: Very nice link, thanks for sharing! I was also thinking of showing off the macro-stepper.

Regarding the lack of libraries: well, Quicklisp is strong of some 1500 libs, which is rather poor compared to most popular languages out there. So yes, it goes both ways!

I'll check out the other libs, thanks!

What I do: Write pure functions, but don't shy away from mutating lexically scoped state in order to improve performance. For global stuff, wrap it with classes since objects are a commonly understood way to reason about state.

For immutable data structures, there is this helper library which can help: http://quickdocs.org/immutable-struct/

I recommend Fset, which other commetners have mentioned.

I wrote a wrapper for it and SERIES, called folio, followed by a revised version called folio2. If your requirements resemble mine when I wrote it, you might findit useful.

You can find it here:

  https://github.com/mikelevins/folio2/

Thanks for sharing!

https://github.com/smithzvk/modf like setf, but doesn't mutate the data (didn't try).

I really enjoy Common Lisp being imperative enough to make iteration and collection orthogonal. You can freely mix-and-match functions for traversing data structures (dohash, maphash, etc) and ways to accumulate values (setf, push, ext:collect, etc.) I find that equivalent purely functional code often needs more functions to both traverse and collect at the same time e.g. fold, map, flatmap, etc.

I know that the functional universe has solutions to these problems, like the State monad and reusable collections protocols based on common minimal primitives, but I really appreciate being able to just freely string together tokens like DOLIST WHEN EXT:COLLECT without any elaborate frameworks.

So I suppose that programming in Common Lisp makes me appreciate non-functional programming.

Having said that, I do feel like a caveman every time I spend brain cycles on picking between EQ/EQL/EQUAL/EQUALP/STRING= or worrying about whether the object I'm updating might come from a quoted constant and invite undefined behaviour. Hard to have it all...

I would far rather have CL's explicit representation of the different types of equality, than many languages approach of either a) "you can't do that" or b) just getting it wrong sometimes. So I would hardly call that behind the times, although the interface could be nicer I suppose.

Assuming that the subtext here is wanting to address potential reasons for Lisp's chronic underpopularity, I think that all conversations like this miss the mark. Lisp was never failing to attract many users because people hadn't experienced sufficient evangelism about all its advanced features. It fails to attract many people because it's not a fun language to get started in.

A year or so back I picked up a copy of Land of Lisp and burned through it for pleasure reading. And I was struck by how gross Lisp looks in that beginner-oriented treatment. Just this huge slog of car and cdr and let/letrec/let* and the 37 flavors of equals and the function namespace. . . and, all the while, you're being told that persevering in mastering this confusing minefield of subtleties will somehow enable you to write bug-free software. I doubt it's actually fun for most people, and the grandiose claims should beggar belief for everyone. I suppose I should count myself fortunate that I got to learn Lisp in college, where there was little attempt to make it fun, and plenty of graded assignments to keep me motivated.

Racket and Clojure are right to clean up some of the language's evolutionary history. That's a start. But even then, the treatment in beginner's guides isn't all that enticing. I've also skimmed through Realm of Racket and Clojure for the Brave and True, and, while both of them work hard at being entertaining (and were fun to just sit and read), they don't really succeed at dispelling the sensation that what you're mostly doing is wrangling with the language itself.

Compare with some of the more popular Python guides. They tend to be much more dryly written, but the actual flow of the guide tends to get you pretty quickly to

    from pypi import have_fun
    have_fun()

First off, Lisp is chronically POPULAR not unpopular. (It is critically unpopular.) Secondly, Lisp is the most fun you can have programming, yeah from the start. You apparently had either a very poor teacher, or taught yourself Lisp ... same thing.

The above post didn't say unpopular, they said 'underpopular'. While that's a bit of a neologism, it just implies that it's less popular than it should be.

Also: "Lisp is the most fun you can have programming, yeah from the start," while subjective, is by most accounts, wrong. Especially when referring to Common Lisp; there could hardly be a more convoluted Lisp than Common Lisp.

Yes, thanks for putting it so concisely.

Personally, I do think Lisp is fun, and was having fun pretty much from the start. But, with how many people I've watched just bounce off of it, I simply can't bring myself to imagine that my experience is anything but unusual. Nor am I inclined to console myself with self-congratulatory stories about Blub.

I suspect that certain styles fit the way certain people think, and don't fit the way that other people think. If Lisp fits the way you think, then learning it is a revelation - it's like being let out of prison. But if it doesn't fit, it's really hard. And those who find Lisp to be "the right way" (for them) expect that if everyone else really understood Lisp, they'd have the same magical experience. When everyone else does not have that experience, they assume that those people just haven't really understood Lisp yet.

I think that this is a function of how you were taught, not a personal preference. An excellent teacher can make pretty much anything interesting to pretty much anyone.

I learned much of my initial Lisp programming decades ago with Macintosh Common Lisp. I found it to be incredible fun to work with a tight interactive environment on my Mac.

So did I--or, rather, I started with Coral Common Lisp, which is what it was called before Apple bought Coral Software.

I found it hugely fun--so much so that it took over my brain. I've been primarily a Lisp programmer ever since.

Coral Common Lisp offered an extremely easy way in to Lisp programming that was also tons of fun to work with, and an industrial-strength development environment for the full panoply of Mac applications. At AAAI in Detroit in 1989 I mentioned to someone that I was working in Coral Common Lisp and a passer by quipped, "Oh, the good Lisp."

It ran just fine on a 1 MB Mac Plus. You could start it up with a double-click. It launched with a Listener window and a blank Lisp source file (unless you launched it by double-clicking an existing Lisp file with some code in it.) You could save an image, copy the image file to a different machine, launch it, and you would see the same windows in the same state.

Creating a fully-functional Mac window looked like:

    (make-instance 'window)

You could populate the window with working widgets with similarly simple interactions. If you built something you wanted to keep using, you could turn it into a Mac application by doing

    (save-application "Foobar")

If you preferred constructing UI by dragging together widgets by direct manipulation, you could do that, too. Once your UI was assembled, you could tell the Lisp to save the Lisp source code needed to reconstruct it.

The built-in editor, Fred, was a species of Emacs, with the Lisp-oriented features that you would expect, but you'd never have to know that if you didn't want to, because it was also a fully-featured Tesler-style modeless editor at the same time.

The stepper, inspector, apropos, and other development niceties all had Mac graphic interfaces that made them easy to discover and tinker with. They were all written in Lisp, so you could use the Lisp reflection tools to crack them open and poke around inside to see how they worked.

Coral Common Lisp made it easy to get started using Lisp to build apps--about as easy at it can possibly be. At the same time, though, it imposed no arbitrary limits on what you could do with it. When I first met Dave Vronay he was working on the GUI for SK8, and gushed about how easy it was to write window-definition resources (WDEFs) in assembly language using CCL's LAP subsystem. He was an experienced arcade-game hacker, well versed in assembler, but Coral Common Lisp was his first fully-interactive assembler, able to define and integrate new assembly-language code while the program under development was running.

The CCL of today, Clozure Common Lisp, is still a great Lisp, but it isn't as easy or as inviting for newbies. It doesn't have the same graphical environment or the tight integration with the underlying system.

In part that's because when the Clozure Common Lisp project was created (under the name "OpenMCL"), its creators had the rights to the compiler and the Lisp runtime, but not to the Macintosh graphical environment. In part it's because modern macOS is quite a bit more complex than Macintosh System 9 and its predecessors, and development systems have a lot of additional hoops to jump through nowadays to do the kinds of things that MCL was able to do.

If a modern MCL existed, I expect a lot more people would find Lisp a lot easier and more fun to get started with. I think it's perfectly possible to create such a Lisp, but it would be a heck of a lot of work.

I used MacScheme earlier, which also was a huge amount of fun - even though I did not have the extension to access the Mac Toolbox (Toolsmith, IIRC).

MCL was a whole new world then. First with Object Lisp and later with CLOS. I used MCL before it was owned by Apple - at one time it was called MACL (from a market agreement with Franz, IIRC). But later LispWorks was another step up, because it was a big grown up full extended Common Lisp with everything from the commercial UNIX workstation Lisps (like Allegro CL, Lucid CL, LispWorks): it suddenly ran on small and simple to use Apple or Windows laptops, plus it had a Cocoa port.

But, as you describe, the simplicity and integration of MCL into the early MacOS was a lot of fun and there was a lot of tinkering by users. Some friends were still using MCL years after it was obsolete...

I started with the full version of MacScheme about the same time that I started with CCL. I loved them both, and at first I liked MacScheme better because Scheme was a smaller, simpler language, and easier to learn. CCL won me over in the end because it was just so much easier to get things done.

Later on, John Ulrich, who owned Lightship Software, the publisher of MacScheme, came to work at Apple and became a friend and colleague. I continued to use MacScheme off and on for years.

I’ve happily paid for Lispworks licenses many times now, but even Lispworks isn’t a good solution for everything I want to use Lisp for. A modern version of CCL’s graphical environment would still be a nicer place for newbies to start. You could even use Lispworks to build a product like that, if the license didn’t prevent it. (Lispworks doesn’t allow their product to be used to make Lisp development systems, which is understandable. They’re a small company who don’t want to put themselves out of business by enabling a customer to give their Crown Jewels away.)

yes, building a product to include development tools (compiler, ...) of LispWorks or Allegro CL would be expensive and not really attractive. I think Franz, Inc. has such a licensing agreement for Allegro CL, but their licensing model is different anyway.

In former times there was a market for add-one GUI or IDE tools. Lucid CL made money from licensing their Lisp to other companies, which then resold those with tools added on (like the SUN Common Lisp IDE). The Action! GUI designer was available for MCL. CLIM was an add-on product. The big Expert System development environments were available as add-on products to Lisp systems.

For a commercial customer something like LispWorks or Allegro CL might still be worth it - they are technically very stable platforms.

I tried to work out a way to reimplement my old programmable WYSIWIG word-processor, Alpaca, using Lispworks, but we've been unable to reach an agreement that works for me and for Lispworks. They don't want me to make something that someone could use to compete with their product, and I don't want to make a crippled version of Alpaca.

I wrote the original Alpaca in CCL, using its Cocoa interface. I used it to write a book, and a Japanese company approached me about licensing it for a product, but I fell ill before I could do anything more with it. By the time I recovered enough to look at it again, it had bitrotted. The macOS interfaces had changed out from under it.

I imagine I could fix it up and get it going again, but I'd really like to make a version of Alpaca that works on Linux and Windows as well as macOS. The obstacle is that I need a rich-text engine that can do a good job with page layout, and I don't know whether there's a suitable one, or which Lisp (that isn't Lispworks) would be best to use with it. Qt maybe? Perhaps with ECL? I guess I'd rather use CCL or SBCL. Anybody know of a text-and-page-layout engine or collection of libraries that will work well on macOS, Linux, and Windows with CCL or SBCL?

Ah. Okay. I misread underpopular as unpopular. My apologies.

Chronically popular? Unless you're using some weird definitions, I would expect to see Lisp used more if you were correct.

The time integral of usage is maybe greater than most languages. Admittedly that’s a conflation of longevity with popularity.

> It fails to attract many people because it's not a fun language to get started in.

Interesting. Nobody is born knowing any computer language. Languages therefore are popular (among other things) in proportion to their ability to convert "those who don't know the language" into "those who know", and "those who know" into actual users. A gentle introduction to those who don't know anything is therefore a good start. I suspect that both Lisp and Haskell suffer from this.

As a professional user of lisp that works with it on a team and delivers products, I agree with the sentiment. Nobody I’ve had the pleasure of working with to on-board has had issues given about two weeks, but that’s in a different environment than a curious somebody who is just poking around from the internet.

The effort to get an editor going and to open a file and load a package (“system” in CL) is a lot. Of course, I’ll die on a hill claiming that those things are a constant overhead and aren’t even detectable when you wield lisp’s power.

Nothing inherently stops lisp from becoming simpler to start with and use, but most everybody who knows lisp has become not only competent with the current tooling, they’re happy with it. I’m happy too: I love using Emacs and SLIME. I love it better than VSCode, PyCharm, etc. I’m faster and more productive.

I wonder why other professions aren’t like programming. Adobe Premier isn’t exactly easy to use, yet professionals aren’t apparently clamoring for an iMovie equivalent.

They’re not? Take a look at at a Final Cut Pro.

Final Cut 10 is a prank, I'm sure of it.

I think that it is even simpler than that.

There are languages for people who want to get things done, and languages for people who want the perfect language for getting things done. The problem is that definitions of perfection differ between people, so people who are searching for perfect languages wind up in a small niche of people who agree, separated from people next door whose idea of perfection is just slightly different.

In the meantime people whose focus is on getting things done go with a popular language which builds a critical mass of ways to get things done. And the availability of useful libraries makes them actually better to get stuff done with than languages that are theoretically nicer in some way.

This is, of course, a worse is better kind of argument. See https://www.dreamsongs.com/RiseOfWorseIsBetter.html for context on that.

I don’t like this argument and paints a picture that Lispers are overly concerned by some academic aspect of programming. Many people, myself included, use Lisp because it lets us get something done more quickly, more efficiently, more easily, or all three.

It’s true. Importing flask in Python and starting a server takes about 1/2 as much code as in Lisp. Some extraordinarily routine stuff Python has down to a one-liner. A lot of people have also written more Python libraries, so chances are you’ll be able to cobble any random doodad together with a huge dependency tree. But after a certain very short prototyping period, I end up fighting Python’s terrible deployment, terrible efficiency, and slapdash language implementation when trying to build something robust.

The following phrase is telling.

"But after a certain very short prototyping period, I end up fighting Python’s terrible deployment, terrible efficiency, and slapdash language implementation when trying to build something robust."

How many programmers would see Python's language implementation as "slapdash" or care? The fact that you both see it that way and care speaks volumes towards you fitting my characterization of the kind of person who uses a minority language.

The other points likewise. Python's inefficiency is well-known and acceptable for most use cases. Where you need better, it lets you escape down to more efficient languages in libraries. Some good examples of this include pandas and various machine learning libraries.

And as for deployment, it is more of an organizational challenge than technical one. The strategy of taking half your servers offline, deploying, then swapping and repeating worked fine 20 years ago. It works fine today. (You might have to do some ceremony with kubernetes. But the strategy remains conceptually similar.) This is only a significant pain point when there are other problems that interfere.

The fact that these things bother you, and bother you enough to rewrite wheels that already exist, speaks volumes.

The fact that people who agree with you on this then can't agree on whether strong typing is worthwhile (like Haskell), or continuations are good (like Scheme) or whether performance is critical (like Common Lisp) or whether access to the JVM toolset matters (like Clojure) causes you to not create a critical mass around a better toolset. And so the situation remains. You all agree that the popular languages are so terrible that you avoid them. But then can't agree on which unpopular language everyone should use instead, or why.

>cant agree on which unpopular language everyone should use

All lispers agree everyone should use lisp. Any lisper that doesnt like lisp is welcome modify that lisp to make it something the lisper does like.

And that is the problem.

Lispers wind up with a myriad of incompatible Lisps, and a solution written for one may or may not be portable to the next. Sure, you can call them all Lisp. But Common Lisp, Emacs Lisp, Clojure, and Racket are, syntactic similarities notwithstanding, actually different languages. A library written for one will generally not work for the others.

Then its only a ten-minute diversion to modify that library to work with my lisp. Or to build a library in my lisp which can work with that library.

I disbelieve.

A complex application written with CLOS will generally not be portable to Emacs Lisp. An application written in Emacs Lisp that takes advantage of dynamic scope won't be easy to port to Clojure. A library in Clojure that wraps something in the JVM ecosystem won't work in Racket. And something written in Racket using continuations or Pict generally won't be easy to port to Common Lisp.

And even with closer Lisps than that, I have enough experience of "that should be easy to port" then blowing up over minor details to be suspicious. Sometimes it will be 10 minutes. Sometimes it will be several weeks and incomplete even then. The latter happens often enough that, the developer's confidence notwithstanding, the average is waaaay longer than the estimated 10 minutes. (The phenomena of tasks being rarely faster than we thought, but roadblocks can be unlimited in extent is generally true in programming. This is one of the reasons why our time estimates tend to be systemically low.)

And if the library is not easy to port, then writing an interface which generates code that DOES work with that library is easy to do.

I still disbelieve.

I've done this. I've seen other people do this. Sometimes it works. Sometimes it doesn't. Sometimes it works fine for the demo but then has weird failure modes that you trip over in production months later.

It may be the right thing to do. It may be feasible. But a general "10 minute job" estimate is laughable.

I can only speak from the perspective of using Clojure, I toyed around with Racket and have no experience with CL.

That said, I disagree with some of the points you make:

> availability of useful libraries

Clojure is embedded in both JS runtimes and the JVM. Both feature massive ecosystems.

In terms of getting things done:

Languages like Python, JS and so on are fantastic at plumbing. But when it comes to modelling data-structures, manipulation and algorithms then they are far less expressive and productive than Clojure from my experience.

I personally think the main issue really is the learning curve. There are quite a number of things that 'suck' about Clojure initially:

1. In a Lisp you are essentially manipulating an AST rather than writing statement-based text. Initially this is cumbersome and taxing both while reading and writing programs.

2. In a functional Lisp like Clojure (and other FP languages) there is a wide variety of commonly used functions which are used to manipulate and compose data-structures and other functions. Especially when reading code this can be daunting.

3. Setting up a environment correctly for REPL driven development as a beginner and tuning it as an intermediate user is quite an undertaking in comparison to many other languages.

4. Clojure specifically being a hosted language forces you to understand the hosted ecosystem as well plus the wiring between the host and Clojure.

The payoff for all those four points is worth it:

1. Search for paredit visualizations like this one: http://danmidwood.com/content/2014/11/21/animated-paredit.ht.... Manipulating code this way requires mechanical prowess and exercise but scales up really nicely, especially alongside better understanding of Lisp code in general.

2. The incredible variety of functional building blocks scales really well with your experience and understanding. You are programming with a series of descriptive expressions, rather than lower level statements. Code becomes more declarative, dense and expressive. Abstraction is much more fluent.

3. A nicely set-up environment enables very fast development cycles and allows for understanding pieces of code of any scope in isolation, because you can evaluate, change and test any expression instantly.

4. This is in my experience a necessary evil to get higher adoption and a the massive library ecosystems of JS and Java.

I assume there are a lot of capable developers who simply cannot get over 1 possibly while being pushed back by 2-4. And I have to admit: If I wasn't already infected with Lisp at a younger age, I probably wouldn't have bothered with Clojure or any Lisp. But now it is my favorite for manipulating and modelling data AKA doing 'business-logic' and transformations between APIs.

Yes, Clojure is a special case. But still let's address your points.

My experience differs. It is hard to find a data structure that cannot be modeled with Python's native data structures with only a constant performance penalty. In fact I believe that I have only seen one, and I don't remember what it is.

I agree with the points that suck about Clojure. To them I must add, "It is hard to hire people who already know the ecosystem." For real businesses this is a non-trivial issue.

As for the purported payoff, I'm glad that you enjoy your editor. For all practical purposes the mechanics of editing code are not a pain point. Besides, I can do the same with vi in any language with braces. (Selecting a block in Python is slightly harder but again has not proven to be a significant challenge.) Designing code as a series of descriptive expressions has more to do with learning to program well than any particular programming language. The third point is not a significant pain point. And as for the fourth point, simply using the hosting language gives you the same library ecosystems without having to understand the wiring and translate all of the examples you find online.

The result is that if you want JS (I only occasionally do), you can just write in JS (or TypeScript). If you want Java (I don't), you can just write in Java.

However Clojure gives you the perfect tool that is customized to exactly how you have to work. Bully for you. My experience with evaluating such environments and tools is that for me to master it is a large investment of time and energy, at the end of which I might or might not (probably not) have much payoff.

(This doesn't stop me from learning said tools. There are good ideas there that I can adopt. But I have not found them particularly useful for me in practice.)

> My experience differs. It is hard to find a data structure that cannot be modeled with Python's native data structures with only a constant performance penalty. In fact I believe that I have only seen one, and I don't remember what it is.

My claim isn't that they don't exist or that one cannot implement them in for example Python (which is very false). I was imprecise there. What I was trying to say is that there is a higher degree of uniformity and generality to do the same in Clojure, which leads to more cleanliness and easier, more fluent abstractions. Clojure code (and I assume similar to true with other Lisps) feels more compatible with itself so to speak. Syntax plays a role here but also the fact that Clojure is designed to be primarily an FP language.

> I agree with the points that suck about Clojure. To them I must add, "It is hard to hire people who already know the ecosystem." For real businesses this is a non-trivial issue. (...)

I fully agree with this and with the rest of your points. I think there might also be factor of personal taste or way of thinking.

I understand what you are saying. I am unable to back it up or refute it from my personal experience. I do not know Clojure well enough.

I do know from personal experience that programming in a familiar environment is significantly more productive than programming in an unfamiliar one. And that many, many programmers have mistaken their personal productivity for the productiveness of the environment.

This is one of the causes of a lot of "holy wars". Because everything from indentation style, to operating system, to text editor, to language feels critical. And indeed is...for the programmer who hasn't learned how to switch between these things.

If someone had an oracle that could give us the answer, I would happily take an even money bet that something of this at least contributes to how much you prefer Clojure.

I also agree with you that there is a lot of personal taste and way of thinking involved. And furthermore that our opinions usually match whatever environment(s) we imprinted on. (In the interests of full disclosure, my background is math, Perl and relational databases.)

I really need to learn CL one of these days; I know Clojure reasonably well, and enough Lisp-Flavour-Erlang, Chicken Scheme, and Racket to be dangerous, but for some reason I seem to have completely avoided CL.

Based on the blog posts I've read, it seems like CL occupies the kind of space I want to be in: sort of the halfway point between theoretical and engineering. Is that a fair conclusion to draw?

It’s a practical language that sometimes chooses less elegant ways to do things in favor of a complete, robust engineering experience. For instance, the fact defined functions sit in a different namespace than values creates mostly aesthetic ugliness where named functions and named values have different treatments.

    (defun f (x) (* x x))
    (setq g (compose f f))
    (g 5)

This is wrong in Common Lisp on many levels: f must be referred to as #'f, and g cannot be called as such, you must use funcall: (funcall g 5)

These might go against the sensibilities one might have had in learning a Lisp in the first place. But in practice, these don’t stop you from writing solid, readable code.

It won’t feel as “clean” or “academic” as Scheme, but you’ll feel it easier to build large and efficient programs without pulling your hair out.

I like it. But the really nice thing about lisp is if there's something I dont like, I can just hack the evaluator to fix it.

If I were doing the above a lot, I'd write compose as a macro (or better yet a compiler macro). In that case, the only thing above that needs to be changed is the setq, which would have to be written as

  (setf (fdefinition g) (compose f f))

Still different from Scheme, but some of the differences can be papered over with macros.

Programming with a lot of higher order functions won't feel as clean in Common Lisp due to the #' and funcall.

Basic Lisp programming with lists is a lot cleaner in a Lisp in which the empty list is false, and in which accessing nonexistent parts of a list (including the empty list) is a safe no-op that yields nil.

Ashwin Ram's (cdr (assq key a-list)) almost works in Common Lisp in the form of (cdr (assoc key a-list)). See: https://ashwinram.org/1986/01/28/a-short-ballad-dedicated-to...

Imperative programming is better supported in Common Lisp because the evaluation of the arguments of most forms, including function calls, is ordered, mainly left to right, so side effect embedded in expressions will show stable, portable behavior. Scheme function calls have unspecified evaluation order, much like C. (In Common Lisp, the only unspecified aspect is whether the function cell is sampled before the arguments are evaluated, or just before the function is called. It's extremely rare for the arguments of a function call to be redefining the function cell, needless to say.)

Common Lisp forms return a predictable value. If it doesn't make sense for a form to return a value, its value is not "undefined", but either just nil or else "no values". When a value returns no values, and an attempt is made to use its value anyway, then nil is produced as the value. You will not see some annoying #<undefined> in a Lisp REPL coming from a procedural construct.

Just mentioning it for historical reasons (someone correct if this is wrong), Rich Hickey was enlightened by CL after doing lots of Java/C#/C++, after doing much CL he created Clojure.

To answer your question, the JVM has wide acceptance in the industry and a big ecosystem of libraries, from a business and practical point of view, it seems the most adequate choice for a Lisp?

Most adequate for what? Peer acceptance?

Are you just purposely ignoring them? adequate for most businesses (the JVM and its ecosystem its going to be accepted way faster than a CL runtime).

I read the comment in its entirety. The adequacy of tools is most often measured by their ability to solve the problems at hand. If the problems are primarily social or legacy, then I could see a completely valid argument that using something JVM-based is most adequate. But one might argue Lisp isn’t adequate at all at that point, and one ought to simply use Java.

Clojure has some other selling points outside of interop though.

For example, Clojure's built-in concurrency primitives like Atoms and Refs are (in my opinion) substantially easier to work with than dealing with "synchronized" or anything like that in Java, and if you're willing to import a library for it, core.async gives you very pleasant Go-like concurrency with channels as well. This allows for a very reactive design that also happens to be safe and relatively easy to debug.

There's also the ability to use macros, which, depending on the use-case, can often replace the need for reflection that you occasionally might have with Java.

Last, it's relatively easy to hook into a running Clojure session with nrepl, meaning that you can use some very-nice runtime debugging and updating without downtime if you're clever.

---------

These features are in most functional languages in some capacity, but most functional languages don't have good interop with Java; the sales pitch for Clojure is comparatively easy: "We should use this language with all these neat features, but where we can use our current stuff just fine".

Yeah, that's why the company I work for lets me use Clojure occasionally; we have a metric ton of libraries written in Java and being able to interop with them more-or-less seamlessly is a huge selling point for Clojure.

For Common Lisp one would use ABCL and probably its JSS library for easier Java calls.

https://abcl.org/

CL compiles down to the metal in most cases (ABCL being a notable exception). I've written device drivers in CL, realtime code in CL, and many CL implementations even allow you to write in assembler. I've done that too, in order to take advantage of SIMD hardware. The above would be -- problematic -- on top of the JVM.

There also might be an issue of speed, since CL is compiled to the metal. Modern JIT JVMs probably have mostly eliminated CL's speed advantage, but I'd still be surprised if some things in SBCL (arguably the best-optimized CL) didn't run faster than in Clojure.

On tools: note that the popular Atom editor has a great extension for interactive CL development, SLIMA, which has all the important features from Slime.

We can also use CL in Jupyter notebooks, see those good examples: https://gist.github.com/WetHat/a49e6f2140b401a190d45d31e052a...

And Vim, Lem, cl-repl, Eclipse (not so interactive)… https://lispcookbook.github.io/cl-cookbook/editor-support.ht...

I learned Common Lisp in high school over a decade ago and have been passively looking for a CL job that doesn't require relocation since. This has been hard (though I did land a few Clojure gigs). Is anyone aware of active Lisp shops?

Amusing set of quotes about CL by famous lispers:

http://xahlee.info/comp/Common_Lisp_quotations.html

A fun read with some good points, although many of them are a bit dated :p

yeah, that was often brought up when he trolled comp.lang.lisp for over a decade.

Common Lisp is still the most fun language to use for me. Sadly lacking in all the currently fashionable languages is interactivity.

> The myth: Common Lisp scripts and binaries are hard to distribute; binaries are too big.

As far as I can tell, it's more like you have to pick one or the other. Distribution is easy if you don't mind distributing a 50 MB image (which is, of course, trivial for some applications and catastrophic for others). Most of the suggestions on reducing application size are essentially accounting sleight-of-hand that doesn't actually reduce the overall footprint (I'm reminded of the compressor that can "compress" any file by one byte... by moving the byte to the file name).

This doesn't scratch my itch at all. I love Lisp but boring old Python has the libraries I need.

EDIT: Based on a comment and downvote I suppose I didn't make my point clear above. I am saying that myths are not the only thing holding Lisp back. (I may be wrong. I hope I'm wrong. Big thank-you to people who are linking ways to get libraries in Lisp, especially Python libraries.)

I care 10x as much about libraries as about all these red herrings (EDIT: or myths, as you like) combined: " * Common Lisp does not have compile-time type checking. * Common Lisp is for imperative, object-oriented programming. * Common Lisp is too specialized, it’s not for general-purpose development. * Common Lisp applications are hard to deploy. "

If you're interested in lisps and python's ecosystem you might enjoy lib python from Clojure: http://gigasquidsoftware.com/blog/2020/01/24/clojure-interop...

https://run.nextjournalusercontent.com/kommen/parens-for-pol...

Thanks! Nothing would be more awesome than Lisp (or Scheme, I don't care) with easy transparent access to modern libraries.

I wonder if someone made an invert hylang.. a python->clojure transpiler

These four things are common myths. It says so in the very line above them. This article is literally about debunking these myths.

Yeah, I read it. Can you explain how "red herrings" differ from "myths"? Is the distinction really that important?

My point was that it is not just myths (or red herrings either) that are holding Lisp back. There are some issues based in reality as well.

I think the distinction is both easy to make and important.

Myths are widely held to be fundamental about a thing (but not necessarily true). Red herrings, on the other hand, are a distraction that draws your attention away from the actual matter at hand.

So that said, I think what you are trying to say is that contrary to the poster, you believe focusing on these myths is a red herring and there are other fundamental reasons that lisp popularity remains what it is. I suspect because you did not articulate the difference, your discussion got sidetracked into semantics because it sounded like you were saying they were "the same thing"

I'm always curious when this comes up: which libraries in particular do you have in mind? If you love Lisp and they're really absent, a short list would at least provide a data point for something that would be desirable and ambitious newcomers could cut their teeth on.

Anyway I don't think the library ecosystem is so dire... Lots of good libraries are distributed through quicklisp. It's pretty straightforward to wrap a C library. (You can even do C++ easily if you switch CL implementations to Clasp...) If you need Java libraries, you can switch CL implementations again and use ABCL. (But of course all your CL code and CL libraries still work.) Lastly, for Python there's https://github.com/pinterface/burgled-batteries (and https://github.com/snmsts/burgled-batteries3 for py3) that even in an incomplete state might suit your particular library needs.

Perhaps this capability isn't very compelling since Python can also access C without trouble (and Java via Jython, .NET via IronPython)? Well I guess all I have left to ask is whether you've considered there might be libraries (or features) in Lisp that would be needed that don't have equivalents in Python? What do you do then? One possible library for admittedly niche applications that came to mind was a hierarchical task planner (https://github.com/shop-planner/shop3) but I forgot someone did indeed make a Python library (https://github.com/oubiwann/pyhop) based on an older version (SHOP1) of the background work, so depending on if you need the v3 features it might suffice.

Thanks for the links! I'll check them out. I like the "higher level" of Python libraries better than C or even Java, because I tend to write what most people would call prototype code. Small user base, small applications, and the really harsh constraint is development time from concept through validation.

No-doubt-unnecessary boilerplate: I understand that people have language wars, and it's possible you may have identified me as belonging to one side or another, but I'll pass. That whole state of affairs is just unfortunate.

In addition, check out py4cl: https://github.com/bendudson/py4cl

(And a Numpy clone: https://numcl.github.io/numcl/)

It's true that are many more for Python than for Common Lisp. Do you have some examples of libraries you are missing in Common Lisp?

Offhand, recently I've used NumPy, telnetlib, openpyxl, TkInter. I haven't used Lisp for a very long time, because of my (perhaps faulty) perception that some of these things (and the next one I will need) might not be there, or might be less usable than their Python counterparts.

> I love Lisp but boring old Python has the libraries I need.

¿Por qué no los dos?

https://github.com/metawilm/cl-python

https://github.com/snmsts/burgled-batteries3

https://docs.hylang.org/en/stable/

http://wiki.call-cc.org/eggref/5/pyffi

I hate the "lists of myths" (that aren't) genre.

> Common Lisp does not have compile-time type checking.

Nothing in the standard mentions compile time checking requirements and there is no useful de-facto standard that you (or tooling!) could seriously build upon either. I'd be suprised if python did not have better "compile-time type checking" for all practical purposes. Yes, SBCL gives much better type warnings at compile time than python, but for python you have mypy and it's A Thing, and still a joke compared to a proper type system like Ocaml's.

> Common Lisp is too specialized, it’s not for general-purpose development.

Common Lisp has no eco-system to speak of for machine learning, web development, command-line utilities, games programming, mobile development, GUI programming, embedded development or pretty much anything real general purpose languages do. If you have something for which Common Lisp is a good fit, you can still be commercially successful using it because not everything requires a super rich eco-system and for some problems what Common Lisp has is in fact highly competitive. But if you want to use common lisp effectively you definitely need to pre-select the problems you want to work on accordingly to an extent that's not true of python, C, C++, Rust, javascript, Go, Java, and half a dozen other languages.

> Common Lisp applications are hard to deploy.

No server or desktop or web-browser comes with common lisp pre-installed. Building common lisp is a pain, because packaging/ASDF is terrible and you don't easily get a nice and small statically linked executable out either. Compared to exactly what language is common lisp not hard to deploy?

>> Common Lisp applications are hard to deploy.

>No server or desktop or web-browser comes with common lisp pre-installed. Building common lisp is a pain, because packaging/ASDF is terrible and you don't easily get a nice and small statically linked executable out either. Compared to exactly what language is common lisp not hard to deploy?

I'm not here to defend all of the flaws in CL, but I really enjoy the SBCL feature that allows you to compile to a binary. Yes it may not be as small as a pure C/C++/Rust executable, but it's very useful and I've used in various projects with great success.

My heart still goes out to Python but I was I could just as easily create a Python executable as I can with SBCL.

Isn't Common Lisp's save-lisp-and-die roughly analogous Pyinstaller? I mean they both work but nobody seems to be using them. In my experience Common Lisp code is most commonly deployed with an interpreter.

Most Common Lisp is actually compiled, not interpreted.

If one compiles code or loads compiled code to something like SBCL and dumps an image, you can both run it directly from that image AND have still have the development tools included.

Some Common Lisp implementations also have delivery tools which create applications without development tools. Some applications though include the development tools, because they are thought to be extended by the user. For example a CL-based CAD system might exposed Common Lisp as the scripting language for CAD extensions.

Of course many implementations do compile to machine code but there are also a few that don't (clisp, ecl, abcl). It is really complicated to talk about languages that have multiple implementations. That is why I went with "interpreter". That was pretty lazy of me :).

But you didn't say anything about the original question. Have you seen Common Lisp code deployed as an executable or as source code?

Interpreted in Lisp usually means 'executing s-expressions by a Lisp interpreter'.

ECL and CLISP also use compilers. ECL usually compiles to C, which then is compiled to native code via a C compiler. CLISP has its own virtual machine, which is the target of its compiler. Don't know what ABCL does, would have to look it up - but in the end it usually would execute JVM code, which gets compiled by its JIT compiler.

Question of code deployment. I have seen deployment as an executable, deployment as an executable with also loads compiled code (for patches) and deployment as source code. Especially commercial applications and/or end-user applications tend to deploy as executable. For example ScoreCloud is a LispWorks application as an executable and can be bought on Apple's Mac app store.

My Lisp Machine at home boots into a Lisp image which contains a whole operating system as compiled Lisp code. That's exotic, but many other Lisp development environments already come as compiled Lisp executable. For example on the Mac and on ARM64/Linux I use LispWorks. On the Mac it is the usual Mac application bundle, where the executable already contains much of the IDE and then loads some code on demand from compile code files.

The word "compiler" is so overloaded that every time someone mentions one it turns into an argument. This is the whole transpiler vs compiler vs interpreter vs JIT argument. The trouble is that sometimes some programmers implement all of them (myself included).

> CLISP has its own virtual machine, which is the target of its compiler

Then CPython is also a compiler? and Lua?

Yes, generally any source to byte-code translator is a compiler. The byte-code then can either be interpreted or compiled, too. Since the default CPython implementation interpreted the byte code, it was said it's an interpreter - but strictly speaking it is not.

Originally interpreters were implementations which execute source code - which in Lisp is widely available. For example SBCL for a long time only had a native code AOT compiler, but now also includes s-expressions.

Famous is also the Lisp interpreter in Lisp from McCarthy, where he defined the core language in itself. See the paper from Paul Graham about that: http://languagelog.ldc.upenn.edu/myl/llog/jmc.pdf That's a very primitive Lisp interpreter - many real ones are implemented in C, Assembler...

Just be aware, that in the Lisp world 'Interpreter' means something very specific: an interpreter of the source language.

Fair enough. My assumption was that you would consider a source to bytecode compiler an interpreter (to be fair most people do). The next time I say something about Common Lisp I will list all the implementations instead of saying something as simplistic as "compiler" or "interpreter".

It's also complicated because some implementations support several modes. ECL for example has a Lisp interpreter, a byte-code compiler&interpreter and a native compilation via C. All code variants can work together and call each other.

> ScoreCloud is a LispWorks application as an executable and can be bought on Apple's Mac app store.

Wow, super impressive. You didn't post this on reddit yet :)

To be clear, ABCL (if I understand correctly) compiles to JVM bytecode, which is analogous to "machine code" in the context of its target platform (and hell, is sometimes even literally machine code if you lack any semblance of sanity: https://en.wikipedia.org/wiki/Java_processor).

And ECL does ultimately compile down to machine code if you include "compile the resulting C output" in the compilation process, though this is optional.

The binary dump feature is used by projects like StumpWM, Lem and Next.

Yeah, I agree save-lisp-and-die is cool. On the other hand you can just write a python script, copy it over and have it run on literally millions on computers out of the box (even accounting for the python3 debacle) – that's pretty handy, too.

Heliohost or Platform.sh come with CL pre-installed (https://github.com/CodyReichert/awesome-cl#deployment).

In my experience, deploying a CL app is muuuch less a pain than a Python one: you build a self-contained binary, you run it and voilà, you can access your web app from the internetz. The binary is ±20MB in size (compiler, debugger and all included). With Python, there are so many ways to fail (and even because a dependency of a library didn't pin its dependencies well enough. Gosh.).

Building a self-contained app is 1 line in the .asd project declaration.

https://lispcookbook.github.io/cl-cookbook/scripting.html#bu...

web development: kind of agree, but that didn't prevent me and two other colleagues to build a commercial website last year, for 1,5 month. It receives tens of thousands of visits a month and has a small admin panel for a three persons team. Job done, rent paid

CLI utils: it's simple to use cl-readline. There are a couple libraries for ncurses. The Lem editor is a good example.

mobile: nope. For the adventurous, see https://gitlab.com/eql/EQL5-Android and its REPL.

GUI: https://github.com/CodyReichert/awesome-cl#gui Qt4, Gtk3, IUP, Tk, Nuklear have good bindings. Qt5 is possible with gobject-introspection (for the adventurous). Proprietary: CAPI. Electron: Ceramic. Java GUI interop with ABCL?

ML: https://github.com/CodyReichert/awesome-cl#machine-learning MGL's author won the Higgs Boson Machine Learning Challenge, but yeah.

for the rest: IDK https://lisp-lang.org/success/

Thanks for the list! :)

addendum: for iOS, Android and mac: possible MOCL (proprietary) https://github.com/CodyReichert/awesome-cl#implementations

CLI utils: you're missing the point. Of course you can write some simple terminal apps with common lisp, but can you name a single one that anyone who is not a lisp fanatic uses? If not, ever wonder why?

GUI: would you develop a commercial app (to pay your rent) in any of these apart from CAPI?

ML: You are joking, right?

I can name one :) pgloader https://tapoueh.org/blog/2014/05/why-is-pgloader-so-much-fas...

I'd use Electron with Ceramic, not the GUI bindings. Or maybe Nuklear. But I would build personal projects with them.

ML: yeah, yeah. Just to show there have been work on it, that (of course) the platform is capable.

Yup, pgloader was also the one example I could think off; I was wondering if you'd name it :) Amusingly I have not been able to get the non-docker version to work due to some bogus openssl problem.

Anyway, I completely agree that Common Lisp is capable, my point also isn't that it's impossible to do things like GUIs, or write something cool and successful with it – just that the eco-system is not general-purpose language quality. If you want to make something that's successful commercially or as a well-adopted open-source project you have to be much more careful what problems you tackle than with a genuinely general purpose language.

Blog posts that claim otherwise really rub me the wrong way; I think it's possible to present CL in a favorable light without distorting reality and no one is gonna thank you later for joining the rank of people who missed out career-wise for making non-viable technology choices.

> not general-purpose language quality.

= not top-notch in all areas? Yeah, there's room for improvement. Buuut, given CL's features and stability, I will very much consider it for a commercial project in the future instead of, say, Python. I'd reject Python, for sure, for GUIs or web dev (except a small website maybe where admins would use the Django admin). Now I must find another language, and it's difficult. Many things drive me back to CL, including its state of libraries, actually pretty good compared to other languages, newer or older. No kidding.

> If you want to make something that's successful commercially or as a well-adopted open-source project

maybe. Reading you, it seems one should avoid using CL altogether. But there's a lot of room for a successful use of CL before falling into these categories: one-off scripts, quick GUIs at work, personal tools? Commercial websites (with adequate requirements), like I did? yes, it's possible to use CL for that. Yes, it is possible to do web development for a job with CL (what kind of sites? We have to ask the few redditors that do it). My point is that CL is general-purpose enough so that it could be more used in the wild (because it is used in the wild).

Anyways. Just trying to be positive :)

> Buuut, given CL's features and stability, I will very much consider it for a commercial project in the future instead of, say, Python[...]My point is that CL is general-purpose enough so that it could be more used in the wild

Yeah, I'm not arguing against that at all. I just hate articles which encourage people towards doomed efforts rather than building out existing areas of strengths or address things that are real obstacles towards using them. I'd like CL to be around for at least long enough till all it's important ideas have made it into other programming languages ;)

For example, one core strength of CL is that it's the about the only really interactive/malleable/live language that can generate pretty performant code, and for sbcl you can hook deeply into the machine code generation as well. Luajit is nicer in some ways (e.g. much smaller footprint in every possible way) but CL has a much more sophisticated interactive development experience (restarts, powerful debugging, pretty-printing etc. etc.). So tooling for automation of binary rewriting, refactoring and hardening like GrammarTech seems to be working on (see https://github.com/GrammaTech?utf8=&q=&type=&language=common...) seems like a much more promising area to work on with common lisp to me than e.g. generic web development.

> Common Lisp has no eco-system to speak of for machine learning,

https://www.cliki.net/Machine%20Learning

> web development,

https://www.cliki.net/Web

https://en.wikipedia.org/wiki/Viaweb

https://en.wikipedia.org/wiki/Reddit#Technology_and_design (though it admittedly got rewritten into Python)

> command-line utilities,

https://github.com/TeMPOraL/hju, among, like, hundreds of other examples

> games programming,

https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp

https://github.com/shirakumo/trial

http://xelf.me/

https://borodust.org/projects/trivial-gamekit/getting-starte...

> mobile development,

https://wukix.com/mocl

http://kriyative.github.io/2011/03/26/ecl-for-ios-updated/

https://common-lisp.net/project/ecl/posts/Lisp-ECL-and-QML-Q...

> GUI programming,

https://common-lisp.net/project/mcclim/

https://www.cliki.net/GUI

> embedded development

https://cliki.net/embedded

http://www.ulisp.com/ (not technically Common Lisp per se AFAICT, but seems to have very similar semantics)

> Building common lisp is a pain, because packaging/ASDF is terrible and you don't easily get a nice and small statically linked executable out either.

https://www.xach.com/lisp/buildapp/

http://www.sbcl.org/manual/#index-save_002dlisp_002dand_002d...

https://ccl.clozure.com/manual/chapter4.9.html

If you think spamming random links to old toy/dead/in-house projects you found on Cliki and that contained words like "machine learning", "Games" or "GUI" (mcclim – really?[]) constructs an effective rebuttal to "no ecosystem to speak of", I'm not sure I know what to say. But I'll go through your list in more detail if you want me to.

[] For the record CLIM had interesting ideas, but last I looked, calling McCLIM's implementation of them a toy would be charitable.

The claim was "no eco-system to speak of". There is clearly an ecosystem to speak of. It might not be a healthy ecosystem, and we might speak of it poorly, but it exists nonetheless, and in fact exists in a greater capacity across that whole range of categories than what a lot of other "popular" languages can claim.

"Machine learning" is a perfect example of how out-of-touch that claim is considering that Lisp was the language for AI research until very very recently.

> mcclim – really?

I guess you didn't see the link right below it with the dozens of other bindings to other popular GUI toolkits.

Very nice overview, I failed to see references to LispWorks and Allegro, though.

Which is a pity because at least for LispWorks, the IDE itself and the CAPI GUI toolkit could be considered killer apps.

I've never used them myself, so in all honesty I have nothing to say about them! :p

That said, I support free software and I believe a proprietary compiler is not a good idea :(

The FOSS experience is a poor imitation of Lisp Machines.

Ideology aside, most proprietary compilers (mostly for Forth, Pascal and Lisp) I have used are of _excellent_ quality.

A lot also provide you the source code once you pay. AllegroCL is one of them I think.

One little thing I did for Allegro was to compile their docs into a dash docset so you can use emacs' dash-docs with it.

- https://github.com/kidd/AllegroLisp.docset

- https://github.com/dash-docs-el/

Lisp(Clojure) best practices: The Pure Function Pipeline Data Flow with Warehouse/Workshop Model

https://github.com/linpengcheng/PurefunctionPipelineDataflow

I don't consider a language supportive of functional programming unless it supports tail-call optimisation, which is performed by some but not all implementations of Common Lisp.

The article recommends SBCL, which does support TCO.

An old (2011) survey of TCO support is at https://0branch.com/notes/tco-cl.html

Indeed, tail-call optimization is not part of the standard, unlike Scheme.

The point of the article is that while the Common Lisp standard is not really focused on functional programming, nothing prevents the implementions (and libraries) of today to be so.

All the CL implementations that matter support TCO.

That sounds more like an implementation detail, in most cases. By this metric Haskell wouldn't be a functional language.

It's a very important implementation detail. Explicit loops tend to imply mutation, which is contrary to idiomatic functional programming. Recursive calls don't require mutation but do require TCO to achieve equivalent space complexity. Constant-factor optimizations are one thing but failing to perform TCO turns constant-space algorithms into linear-space algorithms (or linear ones into quadratic, etc.). It's less a matter of "optimizing" the calls and more a matter of not wasting limited stack space on data which is clearly not required to execute the remainder of the program. One might as well label the practice of freeing stack frames when a function returns "Function Return Optimization" (FRO) and consider it a mere "implementation detail". After all, wouldn't it be much simpler to grab new memory every time the program needs some storage space and never bother with cleaning it up? It would certainly make debugging easier with all those old variables retained for the life of the program and not constantly overwritten by new data. However, programs written for a language without guaranteed "FRO" would look very different from normal programs, much as programs designed to compensate for the lack of guaranteed TCO look very different from idiomatic functional programs.

Haskell uses a different (data-centric, non-strict) evaluation model where recursive definitions don't result in recursive calls, so traditional TCO isn't as relevant. Recursion is used very heavily in Haskell—which has no first-class looping constructs—but the resulting programs generally do not require large stacks. It's not unusual to be able to run even large Haskell programs with a 1 KiB maximum stack size (+RTS -K1k). Space leaks are possible, of course, but they take place in the heap.

Common Lisp was designed without requiring TCO because:

* various platforms don't support TCO. It was designed such that it can be implemented by a simple non-TCO interpreter, transpiled to a non-TCO C compiler, compiled to a non-TCO Lisp Machine CPU, or to a non-TCO virtual machine (like the JVM). Many languages don't support TCO on the JVM and may only implement explicit tail recursion or have a compiler detecting tail recursion - which is far from supporting TCO. Thus a portable conforming Common Lisp program will run in ABCL (a full Common Lisp implementation on top of the JVM) - because it will not depend on TCO to not blow up the stack, or similar.

* another reason was that Lisp has a bunch of features with don't work that well with TCO. For example Lisp always supported various dynamic scoping constructs and made extensive use of those - something which Scheme in its core does not, but provides via libraries or language extensions. Using dynamic scoping constructs makes TCO more difficult, may require a different language design, etc.

I agree with you on the first point. There are good technical reasons why TCO can't be implemented on some platforms. However, that just punts the issue one level down the stack: These platforms should have built in support for guaranteed TCO.

As for language features like dynamic scoping, I would say that a function call which needs to be followed by some cleanup activity is not in tail position, so TCO would not apply. The cleanup code could be in tail position, however, if implemented as a function call. In Common Lisp most forms of dynamic scoping or unwinding are explicit anyway, so this shouldn't come as a surprise as it might in languages like C++ and Rust where destructors are called implicitly when objects go out of scope.

They are often explicit, but they are widely used and often generated behind the scenes by macros or declarations.

Macros would need to specify whether any expressions will be evaluated in tail position. However, expressions in macros don't look like they should be subject to TCO, so the default assumption should be that they aren't unless declared otherwise. Do you have any examples of cases that would be likely to cause confusion—in particular where a function call appears to occur in tail position in the code but can't be TCO'd because of a macro?

For example I see sometimes macros which generate code with compilation quality (speed, ...) declarations for all or parts of their code. Depending on the combination of qualities TCO might be enabled or disabled in code sections.

If TCO is guaranteed at the language level, as in Scheme, then it will always be enabled regardless of compilation settings. Debug builds are no more tolerant of stack space leaks than release builds. The fact that TCO isn't guaranteed is the problem here.

> If TCO is guaranteed at the language level, as in Scheme, then it will always be enabled regardless of compilation settings

https://www.gnu.org/software/kawa/Compatibility.html

Your point? The page you linked to specifically says that Kawa only implements a subset of modern Scheme—by which I mean R5RS or later. Early versions of the Scheme language standard didn't require TCO, but all the recent ones do. This doesn't affect the core point that if TCO is guaranteed by the language standard, as in modern Scheme, then it cannot be selectively disabled because doing so would break perfectly compliant code.

Haskell implements call-by-need reduction, in which all calls are tail calls.

More context:

SBCL supports TCO: see http://www.sbcl.org/manual/#index-Tail-recursion

One can drive whether SBCL optimizes tail calls by setting the proper optimization settings: see http://www.sbcl.org/manual/#Debugger-Policy-Control

Reply on news.ycombinator.com