Common Lisp Runtime Redefinition (2002) | Hacker News Reader

Common Lisp Runtime Redefinition (2002)

tiborsimko.org

71 pointsbrudgers31 comments

What is even more amazing is to start a web server from the Lisp command line (the "REPL"), and then change the definition of a function. The app will use the new function without stopping and/or restarting. And this includes the intermediate step of compiling the whole function!!

You can also update the definition of a class and then force all instances of this class to use the updated definiton... all of this while the program keeps running.

I always tell my friends to try some Common Lisp. The runtime environment is rather amazing. The exception handling system ("condition" system) is also another thing that is impressive on CL. It is so far beyond what Java (or C++) brings to the table, that it isn't even funny to compare them.

> You can also update the definition of a class and then force all instances of this class to use the updated definiton... all of this while the program keeps running.

Right. In fact, you can define a method to be run on instances that were created before the class was redefined, that updates them for the new definition. So if you add a slot to the class, and you need it to be initialized in some way, you can define this method to do what you need.

I have the same discussion every few months, using FORTH as an example. I guess there are a lot of environments where you can use this style of dynamic updates - TCL, FORTH, Lisp, & etc.

And Erlang! Don't forget Erlang :)

Yes, with Tcl starting a server from an interactive interpreter and changing function and class definitions on the fly is trivially simple.

I'm writing a Tcl web sever that uses Tcl's namespace features to change the execution environment the client sees on a per-connection basis. Trivial. Anybody else got that?

I've sometimes used a snippet like this:

    (defun rebuild (&optional (sha1 (sha1-of-current-commit)))
      (asdf:operate 'asdf:compile-op :my-package)
      (asdf:operate 'asdf:load-op :my-package)
      (push sha1 *sha1s-of-loaded-commits*))
    
    (defun rebuild-if-necessary ()
      (let ((sha1 (getf (current-commit) :sha1))
            (loaded (car *sha1s-of-loaded-commits*)))
        (unless (equal sha1 loaded)
          (format t "Rebuilding since ~S differs from ~S.~%"
                  sha1 loaded)
          (rebuild sha1)
          t)))

And then in for example a web service project, I would just call that function in the basic request dispatch handler.

When a web request comes in to the Lisp system, it will check the working directory's Git repository. If the HEAD points to a new commit, it will compile the new code (to machine code, with optimizations!) and reload the whole web app package. When the REBUILD-IF-NECESSARY function call returns, the new code is already loaded, and the web request proceeds as normal, but with the new code loaded.

And yeah, the conditions system is really radically better than exceptions. Raising a condition doesn't unwind the stack, but just calls the closest condition handler, more like a function call than an exception. The handler is offered a choice of restart opportunities, so for example a FILE-NOT-FOUND condition might offer to restart with a different file path, to just retry, or to error. Having restarts available for almost all exceptional conditions makes the system much more flexible.

What I can imagine freaking people out looking at your code is not knowing what, if anything, is atomic or otherwise in a transaction. Presumably that's well-defined by the ASDF operations and whatever multi-threaded solution you're using in Lisp along with the Lisp runtime guarantees, like presumably (I haven't looked) if you compile and load a package through ASDF it will do it all at once (but for each thread?) and you won't get in a state where one thread has a reference to both an old and a new function in the package. You might still have an issue where one thread expects a database schema that no longer applies because you've updated the schema and the mapping code was updated by the package reload but this thread hasn't had that happen yet...

For your web system, are you guaranteeing that if any subsequent requests come in, they wait for the current request to completely finish before being processed? If so, at least there's no obvious bug and the recompile only happens once, but that will turn people away when we're in a many-CPU age. So it's really not that different from the CGI model that PHP has used so successfully. FTP a new copy of the PHP file onto the server, presto, all new requests use that code and you didn't have to restart anything.

Oops, forgot to mention that this is a total hack that I would never put in production. I'm actually very curious about the atomicity of ASDF loads, and I don't know how it works. I wouldn't expect it to be for example atomic with respect to building several dependencies, but maybe it is.

The difference from the CGI model is that the Common Lisp runtime is stateful. In this particular experiment, I was working on a phone menu system, and it was really convenient to have the state of that system as Lisp variables.

You can also do that on Lisp's cousin, Smalltalk. :)

Have you ever delved into Xerox PARC documentation?

Lots of goodies regarding Interlisp-D, Smalltalk and Mesa/Cedar.

Java, .NET and Swift are probably the only modern environments that expose that experience, even if not at the full extent of those environments.

Of course, given the budget, I guess a better option would be to buy a license to one of the surviving Common Lisp commercial environments.

Here's a link for you to give people on Smalltalk hotloading:

https://www.quora.com/What-was-it-like-to-be-at-Xerox-PARC-w...

The impressive part is how Jobs would complain about something in the desktop with Ingalls just changing it on the spot with Smalltalk's rapid prototyping and live update abilities. Key example:

"One of Steve’s ways to feel in control was to object to things that were actually OK, and he did this a few times — but in each case Dan and Larry were able to make the changes to meet the objections on the fly because Smalltalk was not only the most advanced programming language of its time, it was also live at every level, and no change required more than 1/4 second to take effect."

I think people have done this I'm c/c++ with loading shared libraries at runtime. Its just a bit ugly BC the tooling isn't there. But I don't think there is anything inherently magical about it. To oversimplify it, you just have your function table point at a new location with the same signature, no?

Here is an example in C http://nullprogram.com/blog/2014/12/23/ , you can also check the Handmade Hero Youtube series for a more elaborate example.

Please note that the above approach is not as flexible as what you can do with Common Lisp. Also, you will need to use a subset of the language in the case of C++.

You can kind-of do it in C/C++. Unless the function is inlined. The sad reality is that the optimizer screws you over here - either you run your code unoptimized, with all the metadata present (no inlined functions, every function and every method of every class has a symbol, etc.), or you run your program fast. There is no notion of de-optimizing the binary, replacing the definition of a function and then re-optimizing.

You could write that, but it would have to be some kind of hypervisor that combines make+gcc+ld in one package that can dynamically recompile, relink and replace code that changes.

Is CL doing just that?

Also check out https://www.snellman.net/blog/archive/2015-07-27-use-cases-f... for some really interesting, but niche, things you can do to classes.

Runtime definition (and redefinition) of functions was critical to the Lisp Machine's usage model. Since the entire environment was a single Lisp session, everything happened at "runtime", including defining functions. Most development, then, was done by interactively defining functions and trying them out. It certainly was possible to compile a source file into a binary file, and there were occasions when one had munged one's Lisp session to the point that rebooting and reloading the binaries was required, though those were relatively rare.

The combination of dynamicisim, speed, and general utility of the CL runtime is unmatched anywhere except perhaps smalltalk.

With sbcl/slime, I can run a test, get instruction level profiling of the disassembly, make a change and collect profiling data again without restarting my program. The ability to go up and down the levels of abstraction all in the same environment is unparalleled.

Can you not redefine a method in Python's shell in the exact same method?

Edit: Ah...silly me, that isn't exactly runtime.

Right. If I have the Python code to parallel the Common Lisp:

  import time                                                        
                                                                   
  def drill(n, f):                                                   
      for i in range(0, n):                                          
          f(i)                                                       
          time.sleep(1)                                              
                                                                   
  def gen(n):                                                        
      print(n)                                                       
                                                                   
  drill(10, gen)

I get:

  $> python3 interrupt.py
  0
  1
  2
  3
  ^CTraceback (most recent call last):
    File "interrupt.py", line 11, in <module>
      drill(10, gen)
    File "interrupt.py", line 6, in drill
      time.sleep(1)
  KeyboardInterrupt
  $>

The funny thing is though I've known it was possible to redefine running Common Lisp code and I have seen the Continue option many times, it really wasn't clear why it would be used and in part because it is mostly ignored in descriptions of Common Lisp. I mean the behavior isn't described in the SBCL User Manual. Nor is it really fleshed out in the Common Lisp HyperSpec http://www.lispworks.com/documentation/HyperSpec/Body/f_abor...

As a curiosity/hack you can almost get there with pyrasite (https://github.com/lmacken/pyrasite) and

def newfn(): ...

oldfn.__code__ = newfn.__code__

del newfn

There's a difficulty with __globals__ and __closures__ but I found the patch at the end of this post https://hardenedapple.github.io/stories/computers/python_fun... appears to allow full redefinition.

Disclaimer: I'm pretty confident there will be edge cases and/or bugs with these approaches.

Even when it comes to simple function redefinition, Python is __nowhere near__ the flexibility of Common Lisp and that's why things like pyrasite are gimmicks and seldom used in the wild.

The underlying reason is twofold:

+ Python lacks symbols

+ The Python module system is very simplistic

Symbols allow CL to abstract how variable/function access takes place [through the symbol]. Thus, when you change a function's code, everyone that was using the function (through its symbol) will transparently see the change.

In Python there are no symbols but references. When you change the code of the function, you have to manually update all references to point to the new code. If you also take into account the primitive module system that Python throws at you, it gets really messy really fast.

Function redefinition in Common Lisp is part of what makes CL great and the language development environments are geared around it. In sort, Common Lisp developers use it all the time, it's not a gimmick. I never build my full Python programs at the Python REPL, by keeping a single Python process live and iteratively changing its state. I experiment at the Python REPL, sure, but it's so crude and primitive (and the language of course fights you on this by not being geared around interactive development of this sort) that it's impossible to work in the fashion I just described.

When I build a program in Common Lisp, the Lisp process is always live, always running. I am always working inside it and it feels natural and easy.

And of course, this is just one small thing out of the entire 'interactivity' picture when it comes to CL.

The condition system, the object system, the reader, the compiler (available at runtime) and compiler macros ... there is absolutely nothing that comes close to what Common Lisp offers in terms of interactivity today (except maybe Smalltalk which has issues with performance, Forth and to a much more limited extent, Erlang).

I feel that people who point to Python/Ruby/Javascript REPLs or Java/.Net as realistic alternatives when it comes to interactive development and rapid prototyping, have either no experience with or absolutely no understanding of what Common Lisp offers.

> except maybe Smalltalk which has issues with performance, Forth and to a much more limited extent, Erlang

Just to add to the "performance" aspect, Lisp, with modern compiler like SBCL, is generally close or at the same speed as Java (under Oracle's JVM). Which is pretty good.

Just so there's no misunderstanding: I've used CL and I know it's interactive features are better than python. I wasn't saying python is comparable here.

I just want to clarify that the method I mentioned doesn't require finding all references to the original function. It's mutating the function object, so all references will see the updated version.

This isn't a very fair comparison as the Lisp code is shown running in the REPL, not in some sort of release mode that, even in Lisp, would presumably not drop you into the REPL.

It's fairer to ask for interactive mode, in which you can redefine things and restart your tests:

    $ python3 -i interrupt.py
    0
    1
    2
    ^CTraceback (most recent call last):
      File "interrupt.py", line 12, in <module>
        drill(10, gen)
      File "interrupt.py", line 7, in drill
        time.sleep(1)                                              
    KeyboardInterrupt
    >>> def gen(n):
    ...     print('{}!'.format(n))
    ... 
    >>> drill(10, gen)
    0!
    1!
    2!
    3!
    ^CTraceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "interrupt.py", line 7, in drill
        time.sleep(1)                                              
    KeyboardInterrupt
    >>>

What's magic about the Lisp version is not the "in debug mode, you get a REPL and can redefine stuff" part. What's magic about the Lisp version is the "you can then continue the test where you left off" part. That's impossible in Python because exceptions unwind the stack.

Lisp doesn't really have separate release/debug modes[1]. When you want to distribute a program as a binary, you dump the image and set a custom top-level function to use instead of the default REPL. The program will still contain the compiler/interpreter and can redefine everything just like during development (altough the program might not use or expose those features to the user). The REPL isn't in any way magical in Lisp; it's just the default option for interacting with the Lisp.

[1]: You can ask the Lisp to drop all debug information as an optimization, which makes development impractical, but still possible. Some implementation might allow you to distribute a program without the compiler, but the standard does not specify that.

This is why I usually speak of JIT for development and AOT for distribution, as a model.

Lisp doesn't have that in the language standards, but Lisp implementations provide that. Lisp developers had the problem of shipping smaller applications, shipping faster applications to end customers, targetting systems where GC is problematic, or deploying on embedded systems.

Lisps with extensive image-based delivery support are for example Allegro CL and LispWorks. Delivery here means also deep cuts into the software like removing parts of the Lisp system (everything development relatied, unused functionality, ...) and also removing parts of the program for the delivered application. For example the running program might not need much of the program's symbols. Functions would be called directly and not through symbols.

There are also compilers for application delivery (some are inhouse) which generate self-contained C code. These then are compiled with the usual C compiler and generate small static applications. An example would be mocl, which can compile for Android, iOS and macOS platforms.

What you get depends on the error handling. You could also show a read-eval-print-loop on error or make it configurable by turning this on and off. On a Lisp Machine, the user of an application usually would get a dialog with the restart options (something no other typical operating system provides) and one option is to drop into a debugger or an error read-eval-print-loop. For some commercial systems this could be hidden from the user.

Far as restarts vs crashes, Burroughs MCP could do something similar albeit not as impressive or flexible:

"To accomplish such enhanced protection, a newer mechanism was introduced in the mid 1990s. In an unfortunate and misguided attempt at compatibility, it was named after the then-proposed C++ language construct of the same name. Because the syntax and behavior of the two differ to such a large extent, choosing the same name has only led to confusion and misunderstanding.

Syntactically, 'try' statements look like 'if' statements: 'try', followed by a statement or block, followed by 'else' and another statement or block. Additional 'else' clauses may follow the first. During execution, if any recoverable termination occurs in the code following the 'try' clause, the stack is cut back if required, and control branches to the code following the first 'else'. In addition, attributes are set to allow the program to determine what happened and where (including the specific line number).

Most events that would result in task termination are recoverable. This includes stack overflow, array access out-of-bounds, integer over/under flow, etc. Operator (or user) DS is not recoverable except by privileged tasks using an UNSAFE form of try.

MCP thus provides a very fault-tolerant environment, not the crash-and-burn core-dump of other systems."

> This isn't a very fair comparison as the Lisp code is shown running in the REPL, not in some sort of release mode that, even in Lisp, would presumably not drop you into the REPL.

This is not correct. There isn't such distinction between "release mode" and "debug mode"; the runtime -which includes the repl, interactive condition debugging and other nicetimes- is always available. It is not a thing you only get "in debug mode".

What you can do, if you want, is to enable some optimizations and drop some debug info. For example you can put, on your code:

   (proclaim '(optimize (speed 3) (safety 0) ( debug 0)))

... and this will tell the compiler that your code wants ludicrous speed, that you don't care about safety or additional debug info. But you will still have the repl and everything.

I didn't say the REPL wasn't available, I said dropping you into the REPL would not be the default action on an unhandled condition. Others have confirmed that this is the case.

Thank you anyway for your remarks.

Reply on news.ycombinator.com