Running Lisp in Production (2015) | Hacker News Reader

Running Lisp in Production (2015)

tech.grammarly.com

138 pointstosh68 comments

I like lisp as a programming language.

But, this article gives one of the reasons for avoiding its use in a high performance production environment. Provisioning a VM with over 25GB RAM for an application which requires about 4GB of it is a huge waste of resources.

While the common wisdom on HN is that for most applications, developer costs more than physical resources. Physical resources still cost money.

Initially list might be a good option for development velocity. But, over time as the application matures, adding features becomes more complicated and the velocity goes down anyway. Shifting to something which reduces costs by upto 6x might be worth it.

This is not so much a Lisp problem as an SBCL problem. SBCL just doesn't have a very good garbage collector. I nearly gave up on Lisp fighting with it, but ultimately switched to Clozure instead, and never regretted it: Clozure has a fantastic garbage collector. I run Clozure and Postgres side by side on one 4G Linode without a peep.

I thought you misspelled Clojure initially (I guess it does also benefit from decent gc). I didn't know about Clozure but will investigate when I next use CL.

Also referred to as CCL. It's a good implementation. The compiler doesn't generate as efficient code as SBCL does, and AFAIK doesn't do as much type inference as SBCL, but you get much faster compilation, and the generated code is still quite performant - I used CCL for hobby-level gamedev ~6 years ago, back when CCL was the only implementation that would run on Windows reliably and fast enough for games. These days, SBCL works on Windows too, so that's what I use most of the time - but then, CCL becomes helpful on Raspberry Pi to use all CPU cores, as SBCL (AFAIK, as of 1-2 years ago) still doesn't do proper threading on ARM.

I wrote some high-performance lisp applications in production. My main was was a threaded queue worker, driven off beanstalkd, that processed large volumes of data. This was highly successful and didn't require overly-provisioned servers (was using CCL/linux x64/linode VPS).

That said, I don't think I'll ever use lisp for front-facing web stuff again. I wrote cl-async, and afterwards and used it in production in a few places, then a bit later started using node heavily. The tooling for CL is just not there. Quicklisp is great, but not being able to pick and choose package versions is limiting, and having to rewrite every single thing for async was a pain (since CL doesn't have continuations, any sort of async that avoids CPS is impossible). I was hoping I would get more community support on cl-async, and I did get many hands helping, but eventually it just became too much to maintain. So if you're doing network stuff on CL, it's probably best to just use threaded, which IMO just doesn't handle the same amount of load evented does, at least when dealing with web APIs.

One thing I used lisp for briefly which I want to do again soon is game dev. Being able to redefine/fix portions of your program while it's running is fantastic, especially when you have huge amounts of state to deal with like you do in games. I wouldn't exactly call that running CL in production, though.

FWIW, Qlot solves the version pinning problem. Not to discredit your experience, just wanted to say that there are solutions today for the same issue.

I'm interested exactly what the pain points are with CPS. Do you mean manually rewriting in continuations is tedious or is there a more insidious issue. I haven't tried writing async code in Lisp but macros would appear to eliminate continuation-writing-by-hand.

> macros would appear to eliminate continuation-writing-by-hand

They don't =]. There's cl-cont, but it's like building a house on a foundation of toothpicks. Not to discount the work the author put in, but you can only do very specific things with it, and last time I looked, error/exception handling was pretty much completely missing.

> I'm interested exactly what the pain points are with CPS

Basically, for example, if I'm going to convert cl-postgres to use async, I have to rip out the protocol parsing from the networking (they are often globbed together) then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up. This can be somewhat alleviated by promises (https://orthecreedence.github.io/blackbird/) but they still don't change the fact that the entire stack is unwound once you call into async.

Continuations would allow building green threads/coroutines which would eliminate CPS and allow you to preserve your stack while calling into async, but CL is pretty much written in stone to not have continuations (or coroutines et al) at this point.

Qlot looks handy, thanks for the tip.

> then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up.

Isn't this the very consequence of going async? I'm reminded of the article on this recently featured here[1]. Once you introduce async, you poison everything up to the event loop (which, with usual async implementations, you don't even see or control).

[0] - http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... + https://news.ycombinator.com/item?id=16732948

> Isn't this the very consequence of going async?

Like I mentioned, depends on the language features. If it has coroutines/green threads/continuations, you can use async without unwinding the entire stack. CL will probably always have this problem, unless the bigger implementations independently decide to implement the necessary building blocks (which they probably won't).

Try explaining that to me 5 years ago, though.

If it's about avoiding stack unwinding, then maybe you could hack an implementation with the condition/restart system? This is the one stack-preserving feature in CL.

Biggest issue I personally had with CPS and why I'll never use it for anything is the need to wrap your entire call stack, at compilation time, in CPS macros. That's a no-go for calling external libs.

Clojure does this a lot better with Pulsar.

Thanks for your work on cl-async/blackbird: we also used it in our recent project (and you should have gotten a couple of related requests)! I don't think you should be discouraged though: your contribution is very valuable and surely was noted, just that the whole async approach is, in my view, not in line with the general direction of where Lisp is most applicable, currently (we should admit that it's a niche language). After all, having an evented server means that it doesn't perform a lot of useful computations, most of the work is just serving some kind of existing content. I.e. you could say that the application logic is marginalized and commoditized. And Lisp is the best choice for the opposite cases, those that require and emphasize elaborate application logic and intense computation...

I use cl-async and want to thank you for it.

Anything further from your experience and retrospective working on the project would be invaluable to document somewhere for the CL community at some point. Just your insight here in the above comment is very interesting. Over time things may improve.

Thanks for the nice note! I'm glad you still find it useful.

I do hope things improve over time. I'd love to come back to CL, and actually plan to sometime soon. I'd like to get my pet project, Turtl (which was using cl-async/Wookie in production and is now migrating to node), in a stable place (and hopefully making money from it at some point) and then start dipping my toe into game programming again. I'll almost definitely use CL over anything else. Maybe I'll write the GPG-verifying, version-pinning quicklisp wrapper I've always wanted while I'm at it =].

As far as retrospective, I think I suffered from the part of the lisp community I had always hoped wasn't true (or hoped I could change): people are much quicker to build their own version than to improve on something existing. I think if Wookie had gotten more attention and cl-async had had a few more drivers written (postgres, mysql, etc) it could have gotten a foothold. It didn't though, and I got tired of hacking through the digital reeds with my machete when I already knew javascript pretty well and node had all this stuff figured out already. It's funny, after using Express for ~4 years, I keep having the urge to go back and implement what I've learned into Wookie. I have to tell myself that it's probably best to just let it sit for now.

On another note, my dog, Wookie, who the project was named after, died last week. So it seems even more appropriate to let the whole thing rest. I might re-use the name if I come back to the CL world, though. Seems its doppleganger project got more focus anyway.

About picking and choosing package versions; isn’t it straightforward to keep whatever versions of packages you want under ~/quicklisp/local-projects ? Or is there a need for a more sophisticated tool for this?

This works, sure, but when setting up a new server, you're going to have to remember which package version you want for each of your dependencies. It's much easier to just have a Cargo.toml or package.json and run one command that verifies and installs exactly what you need.

I guess containerization solves much of this problem now, but back when I was heavily into CL, containers were just starting to peek their heads out into the world at large.

Solaris and openvz containers have been around since 2005 or so.

Solaris is not that accessible to most people the way Linux is. And containers are overkill when all you need is the ability to put things in a sub-directory.

I'm a little bit disappointed that you've decided to stop maintaining cl-async, although I appreciate the difficulty of maintaining such a project.

Hopefully it's still useful! The nice thing about CL is that it never fucking changes so you can write something and 25 years later it still runs. Really, the only moving targets are the bundled ASDF version implementations ship with and libuv itself. I've had some reports of cl-libuv not compiling against the latest libuvs, but that shouldn't be too hard to fix (I still merge PRs =]).

All in all, I think I just hit one too many problems using cl-async/wookie in production to justify using it vs Node which just kind of works and has a plethora of maintained packages.

I don't necessarily disagree with you, but provisioning 16-32gb for a production server doesn't seem to be unusual these days and the cost negligible...

The cost is negligible for 1 server.

For 100s to 1000s of servers. The cost starts mattering.

I'm not saying new applications shouldn't be built on lisp. Most applications don't have huge scaling requirements. But, if you expect the application to grow, have a plan in place for migration.

>For 100s to 1000s of servers. The cost starts mattering.

If you need 100s to 1000s of servers you presumably already have customers and money to spend.

To spend on rewriting the application in another language.

AKA a "nice problem to have" ("oh, we're now so successful we have to rewrite the app in another language to lower our infrastructure costs".

Most of the time you aren't gonna need it (because you wont go far anyway as a company). And when you do -- like Twitter and Facebook did, you'll be swimming in users and money already anyway...

> Provisioning a VM with over 25GB RAM for an application which requires about 4GB of it is a huge waste of resources.

Like pretty much every JVM-based application ever? For the last 15 years?

Not according to the article. They say that SBCL's GC is not in the same league as the JVM's GC, based on experience running other services on the JVM.

Are you saying that's not true, and that the JVM is actually no better than SBCL? I'd love to read a counterpoint.

JVM's GC is surely better (actually, is there any other platform that has a better-developed GC infrastructure?) But the point, I guess, was not about the GC but about the usual size of Java applications

I am not saying that, and that's not implied by what I was replying to, either. The comment is about allocated heap sizes differing from the application's actual memory usage. That's pretty typical of JVM applications, and a source of consternation for my sysadmin friends.

Yes, when Elixir runs amazing on a $5 a month digitalocean account.

From the article:

> Our Lisp services are conceptually a classical AI application that operates on huge piles of knowledge created by linguists and researchers. It’s mostly a CPU-bound program, and is one of the biggest consumers of computing resources in our network.

CPU-intensive processing is exactly the kind of work that the BEAM is lousy at handling. Elixir would be a terrible implementation language for this kind of thing.

My Lisp web-app does too. I barely get any traffic, but still.

well, those 25 GB come bundled with the number of hyperthreads that we used (i.e. 8 for an AWS xlarge server), so we didn't bother to optimize for memory usage

I wonder if Chicken Scheme has the same memory problems as SBCL. It seems to me that it shouldn't.

Chicken itself is very lightweight. Applications built on it may or may not be.

It also has a fraction of the features

In Chicken, those features might not be in the core language but in the libraries. It really depends on which features you need. If they're not in Chicken or its libraries, ask yourself if they're really worth the bloat of SBCL?

Also, Chicken offers a number of other advantages apart from a much smaller footprint, like elegance, a less crufty and more modern syntax, and the ability to compile down to C.

That the syntax of Scheme (from 1975) is more modern than the syntax of CL (from 1984) is a myth. It's more primitive. Later people added lots of features to Scheme, which were already in CL. Take keyword arguments... In essence Scheme is actually less elegant due to its too small design and the many bolted on features.

Chicken:

    ((lambda (x y #!optional z #!rest r #!key i (j 1)) 
        (list x y z i: i j: j))
     3 4 5 i: 6 i: 7)            
    => (3 4 5 i: 6 j: 1)

CL:

    ((lambda (x y &optional z &rest r &key i (j 1))
        (list x y z :i i :j j))
     3 4 5 :i 6 :i 7)
    => (3 4 5 :I 6 :J 1)

Is the CL code more crufty compared to the almost equivalent Scheme code? Sure not. or is #!optional now more modern than &optional?

Guess where the Chicken feature originally came from ...

The difference: the CL code works in every CL implementation.

> and the ability to compile down to C.

There are many implementations which do that. SBCL has the ability to compile to optimized native code with a type-inferencing compiler...

Chicken is a great implementation with a nice ecosystem, but that it is more elegant than CL - while it copies lots of features from CL (like COOPS copies from CLOS) - is an illusion.

http://wiki.call-cc.org/eggref/4/coops

> COOPS provides classes, generic functions and methods, similar in style and use to the classic Lisp object systems like Flavors, Loops or CLOS. For general information about object-oriented programming in the context of Lisp, consult one of the various books and guides available to the subject.

Example:

Chicken:

    (define-method (pop (stack <stack>))
      (let* ((c (slot-value stack 'content))
             (x (car c)))
        (set! (slot-value stack 'content) (cdr c))
        x))

Let's see how it's in Common Lisp:

    (defmethod pop ((s stack)) 
      (let* ((c (slot-value s 'content))
             (x (car c)))
        (setf (slot-value s 'content) (cdr c))
        x))

Which is more modern? To me it looks the same. The difference: the CLOS code runs in every CL implementation.

Also:

   (defmethod popm ((s stack))      ;; don't clash with cl:pop macro
     (pop (slot-value s 'content))) ;; speaking of which, use it.

I think you have to write your own pop macro in Scheme or find some lib.

> to compile down to C.

This HN submission's referenced article ("Running Lisp in Production" (2015)) is about SBCL which compiles down to native code.

There are CL implementations that compile to C, if you think that is better for some reason.

For instance, derivatives of Kyoto Common Lisp, such as GNU Common Lisp.

Or ECL:

https://common-lisp.net/project/ecl/static/manual/ch34s02.ht... "The other type of compilation is the so-called "native" compilation. This process consists on translating the lisp source file to C language. The intermediate file is later compiled using a C compiler."

You've never run a Tomcat from 2009 haven't you?

What would that something be?

Most "mainstream" languages aren't any better.

Something with a large ecosystem and community that has ready-made solutions for tuning, deployment, and integration with other products and workflows. Java and JavaScript are front-runners. Python is good too.

If you have to build rather than buy all the infrastructure that's not your core product, you lose. That's a huge yak to shave, and startups can't -- and mature businesses won't -- absorb the upfront costs of doing all that work.

"Java and JavaScript are front-runners."

Which is where Clojure comes in.

you misunderstand the meaning of startups. Startups, at core, are not about rebuilding the same stuff by combining existing solutions but about building novel solutions to solve yet unsolved problems. The other things are really secondary to that

Indeed. As a startup ypu want to build as little as possible tgat's already been built -- so you want the language/framework that has the biggest most accessible ecosystem, so that you can focus on building things unique to your line of business.

That's not unique to startup - I'd say it's even more relevant to regular "corporate" software development, no? What's unique to real startups (for my definition of "real": the ones that solve yet unsolved technically challenging problems) is: even though you may be better off using a good ready-made infrastructure, that doesn't change the game for you as your competitors also have access to it. While using a conceptually different (more powerful?) toolset might give you an edge over the competitors and a chance to indeed solve the problem you're after. Just yesterday I talked about this: https://www.slideshare.net/vseloved/lisp-in-a-startup-the-go...

> While the industry waits for LightTable and similar tools to mature, Lisp programmers have been silently and haughtily enjoying such capabilities with SLIME for many years.

Were a significant number of Lispers really waiting for LightTable to be usable for them?

Clojure guy here, FWIW

The big deal with tools like LightTable is it's much easier for newcomers to start working on projects. I understand why a user wouldn't want to learn both a language and an editor. One huge issue we struggle with emacs/slime as the editor of choice. Personally I'm a VIM guy, so I got it working there, but that's after I spent some time deep in Vimscript and fiddling with plugins. It's a lot to ask for people that are new, or those that have an editor of choice that isn't supported.

Did you know emacs has several vi implementations? Many people think they are the best versions of vi.

Except for lisp -- emacs vi implementations don't usually play as well with parenthesis as native emacs, which is superb for lisp coding.

you're wrong - Spacemacs offers very nice tools for structural editing out of the box and including clever-parens layer. You can also use Parinfer or Lispy. You can customize things to your liking using combination of Paredit and Smartparens.

Only tried evil (which I think is the Spacemacs default?) with paredit and while paredit is a truly stellar, evil mode does not play well with it. Perhaps the other paren tools you've mentioned work better with evil? With paredit, evil editing can delete only part of a paren leaving unmatched problems.

paredit + evil-mode is really fantastic

Except that evil mode takes priority over paredit and it is quite possible to delete parens and end up with unmatched.

That’s why you add evil-paredit-mode to fix that

been there, tried it

I doubt many Lispers were because we already have emacs and SLIME. One of the most fun things I did was running a window manager called stumpwm and running swank in that image so it could be modified on the fly with emacs. That's also how you modify emacs itself. It's just so cool to use a tool to modify itself while it's running...

If so, they will sadly be waiting forever, as LightTable is largely dead.

The creator, Chris Granger, abandoned it for his next project, Eve. After it became clear Chris had left, the community tried to pull together and continue, but it looks like it hasn't been updated in months.

Presumably "the industry" ≠ "lisp programmers"...

Almost all clojure ide I have used have similar capabilities to lighttable (emacs, atom, sublime, intellij with plugin) because they know how to talk to the nRepl

> Fortunately, this obscure Stack overflow question helped [https://stackoverflow.com/questions/442556/spec-for-junit-xm...] — we ended up having to build this into our own testing library should-test.

> closed as off-topic by kleopatra, Martijn Pieters Jun 19 '15 at 19:55

All the best SO questions get closed.

And the reason given for closing it demonstrates zero understanding of the question being asked.

"Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam."

The question is asking how to find the specification for an XML format. This is very much a "only one correct answer" kind of question, not a "which X do you like best?" kind of question.

Seems completely justified to be closed. Although this does seem to fall in scope as it would be very difficult to locate without knowing exactly what you are after.

Which is why it's probably fair that they're closed but not deleted.

There really does need to be a tag for that kind of question.

Something along the lines of I'm looking for a spec about X, but I haven't had any luck finding it.

It's a bit of a tricky one as it may get downvoted / closed, but another approach is to try and avoid asking for the resource and ask something like

What is the specification for X?

What is the name of the specification for X?

Is there a specification that defines the output of X?

While most of the answers will likely link to a resource, technically you didn't ask for an offsite resource, however it may be seen as too broad , or still may get closed for asking for offsite resources.

To avoid making it too broad, don't ask "what is the specification of X" where X is the whole darn can of beans. Ask about some small example instance which falls under the X definition: what is the exact meaning of this Y object/syntax/whatever, that can be input to system X?

Reply on news.ycombinator.com