Emacs Lisp Readable Closures (2013) | Hacker News Reader

Emacs Lisp Readable Closures (2013)

nullprogram.com

106 pointsnanna29 comments

For comparison, since I'm using Emacs with native-compilation:

Interpreted result:

  ELISP> (hn/foo :bar :ignored)

  (closure
   ((y . :ignored)
    (x . :bar)
    t)
   nil x)

This is equivalent to the output in the article. Now, after running compile-defun on the function and calling it, I get:

  ELISP> (hn/foo :bar :ignored)

  #f(compiled-function
     ()
     #<bytecode 0x1f40000a8a1d>)

This is the byte-compiled case, which sadly is not readable for some reason. That said, IELM (Elisp REPL) makes that #<bytecode ...> form clickable (it's a so-called "presentation"), which leads straight to the following disassembly:

  byte code:
    args: nil
  0       varref x
  1       return

After native-compiling via (load (native-compile "file.el")), I get the same output as with byte-compiled case (+/- the address in #<bytecode ...> form), but the disassembly is slightly different:

  byte code:
    args: nil
  0       constant :bar
  1       return

EDIT: turns out it's IELM that's playing games with the printer. If I write:

  (print (hn/foo :bar :ignored))

instead, I get the readable output from the article, in both byte and native-compiled cases. My understanding of the latter is, the closure returned from a native-compiled function is itself only byte-compiled (at least when used in REPL), and the difference in bytecode assembly come from the latter running through a more thorough optimization process.

So how does this interact with updating the captured environment? Does the printing essentially just show a snapshot of the environment at that point?

I assume the (closure) is read back as a new closure such that the environment will be restored as an environment shared between invocations of the read result?

> I am unaware of any other programming language that has this feature

S7 scheme:

  > (define f (let ((x 5)) (lambda (y) (+ x y))))
  f
  > (f 4)
  9
  > (object->string f :readable)
  "(let ((x 5)) (lambda (y) (+ x y)))"

There is even talk of printing continuations readably.

IIRC the python-on-guile project can probably do this using some C voodoo. I believe it can serialize continuations and make deep copies of generators with mutable state.

Edit: however, I doubt the serialized format is human readable by default...

This isn't really the same, though. You are printing f. Print the closure returned by f.

This is scheme. f doesn't return a closure; it is a closure, and it returns a number.

Still feels different.

For example, this

    (let ((x 3))
      (let ((y (lambda (z) (lambda ()  (+ x z)))))

        (message "%s" (funcall y 3))
        (message "hello")
        (setq x 5)
        (message "%s" (funcall y 3))))

will produce the following messages

    (closure ((z . 3) (x . 3) t) nil (+ x z))
    hello
    (closure ((z . 3) (x . 5) t) nil (+ x z))

While this:

    (let ((x 3))
      (let ((y (lambda (z) (lambda ()  (let ((w 0)) (+ w x z))))))

        (message "%s" (funcall y 3))
        (message "hello")
        (setq x 5)
        (message "%s" (funcall y 3))))

will print:

    (closure ((z . 3) (x . 3) t) nil (let ((w 0)) (+ w x z)))
    hello
    (closure ((z . 3) (x . 5) t) nil (let ((w 0)) (+ w x z)))

Is it just that emacs is more explicit about which values were closed over?

Edit: Actually, I see my confusion, you accidentally printed 'f', not 'g', which I think was your intent?

> Edit: Actually, I see my confusion, you accidentally printed 'f', not 'g', which I think was your intent?

I am not quite sure what you mean? There was no 'g' at all in my code snippet except for that in 'object->string'. Unless you are mistaking the '9' for one?

Is what happens when I write so late at night. I mistook the 9 for a g, thinking it was showing the body of the lambda. Really, I was just reading it all sorts of wrong.

I still feel this is somehow different, but I can't express the difference well.

Note that I should say I am not at all surprised that scheme could do this, for what its worth. I'm just saying that this feels different.

Here are your snippets in s7, and their output.

  (let ((x 3))
    (let ((y (lambda (z) (lambda ()  (+ x z)))))
      (format #t "~a~%" (object->string (y 3) :readable))
      (format #t "hello~%")
      (set! x 5)
      (format #t "~a~%" (object->string (y 3) :readable))))

Which gives:

  (let ((z 3) (x 3)) (lambda () (+ x z)))
  hello
  (let ((z 3) (x 5)) (lambda () (+ x z)))

And:

  (let ((x 3))
    (let ((y (lambda (z) (lambda ()  (let ((w 0)) (+ w x z))))))
      (format #t "~a~%" (object->string (y 3) :readable))
      (format #t "hello~%")
      (set! x 5)
      (format #t "~a~%" (object->string (y 3) :readable))))

Which gives:

  (let ((z 3) (x 3)) (lambda () (let ((w 0)) (+ w x z))))
  hello
  (let ((z 3) (x 5)) (lambda () (let ((w 0)) (+ w x z))))

Cool, so it is just more explicit it elisp. Curious if there is a reason for that.

And thanks for running these! I can't really give a reason for not installing a scheme to try myself... :(

R can do this too if I understand correctly.

Even though not part of the language, Clojure can do this via a library like https://github.com/technomancy/serializable-fn (I just learned about this).

You can serialize closures (and continuations) in Common Lisp using Common Cold. The serialized representation of a closure contains a tag instead of the actual code, but the forms are also available in case you need to serialize them as well.

  CL-USER> (write-to-string
            (let ((y 4))
              (common-cold:slambda (x)
                (* x y))))
  "#.(S:F '1 '(4))"
  CL-USER> (funcall (read-from-string *) 3) ;; * here means reference to the last returned value
  12

> The serialized representation of a closure contains a tag instead of the actual code

Was going to ask about that.

For readers unfamiliar with Common Lisp, the output is essentially equivalent to

  "(eval '(s:f '1 '(4)))"

because #. is a reader macro that means "instead of returning the s-expression, return the result of evaluating it". I.e. it's a way to run arbitrary Lisp code at parsing stage. This is not something you want to see in a serialization format.

Ignoring the security implications[0], the result seems to rely on the runtime state of the Lisp image (I assume that '1 bit is a key to the actual code of the function) - so you won't be able to persist it and load after program restart, or send it to another instance of the program.

[0] - We're talking serializing code here anyway, so adding arbitrary code execution to paring changes nothing.

Like I said, the mapping from tags to forms is available and you can serialize that as well, if you wish. Using a tag is useful, as it makes for a compact serialization when creating many closures of the same function. With regards to security, you would want to attach an authentication tag that you can verify prior to deserialization. Common Cold comes with an example of a Hunchentoot handler that compresses and encrypts the serialized continuation representations.

Thank you for sharing this!

I tried looking for this on QL and github but couldn't find it ... do you know if the common cold source code is still available somewhere?

I believe you can find it via archive.org, but you should probably ask Paul Khoung (pvk.ca) to push it to GitHub or something.

You're right, it was: http://web.archive.org/web/20081120155011/http://www.discont...

It’s such a shame that the Common Lisp community has to rely on archive.org so much. I remember working with others to recover (IIRC) https://wiki.alu.org/ from archive.org after... something.

In fact, it loks like it’s inaccessible now, at least in my browser. Google does have some pages indexed though: https://www.google.com/search?q=site%3Aalu.org

Yes, it is unfortunate. Information loss on the internet is not limited to CL, however, and many of my bookmarks became stale over the years. When I encounter a "treasure trove" of information, I make sure to archive it on my machine (and ultimately on backup hardware). Sometimes I put it up on sites like GitHub.

Wow, this is a really cool feature... specially the compiled lambda form with readable bytecode.

I will have a look at the "next" article from that post that explains emacs bytecode in more detail, I can imagine that opens the door to really interesting features in emacs.

How does this work if the closure will mutate a closed-over location? e.g.

  (let ((x 1))
   (funcall (read (prin1-to-string (lambda () (setf x 3)))))
   (print x))

[edit]

Realized I have Emacs installed and tried it; the above prints "1" while calling the lambda without first serializing causes the (print x) to print "3"

Python Dill, https://github.com/uqfoundation/dill can serialize closures.

From 2013, but interesting, and lots of other good Emacs-related stuff on that site.

Seconded. The author is the creator of the magestic Elfeed.

and has a talent for short enough yet deep enough articles

Zenlisp by Nils Holm has readable closure too. http://t3x.org/

Zenlisp on Github: https://github.com/barak/zenlisp

Wow he’s written lots of books. Can you recommend any?

Reply on news.ycombinator.com