The Common Lisp Condition System – Upcoming Book
apress.com
apress.com
The book happened because I got annoyed at the fact that there is no good reference that shows how the condition system works, what are the foundations required for it to work, what are the basic building blocks that it is composed of, and what are its uses, both in the domain of handling exceptional situations and, more importantly and less obviously, outside that domain.
The book also contains a description of a full implementation of a condition system that can (and hopefully will!) be used outside Common Lisp. There's hope that some languages will pick up these parts of Common Lisp that fully decouple the acts and means of signaling conditions from the acts and means of handling them, therefore allowing for means of dynamically controlled control flow that are currently very hard to achieve in other languages.
The accompanying code is available at https://github.com/phoe/tclcs-code and https://github.com/phoe/portable-condition-system - it will be copied over to the Apress repository soon.
Linked Reddit thread: https://www.reddit.com/r/lisp/comments/hrjzs8/
AMA, I guess.
As for ToC, hold on, give me a few minutes - I'll extract it for you from the manuscript.
EDIT: ToC is here, https://plaster.tymoon.eu/view/1956
The book explicitly contains a tutorial section that describes dynamic variables, non-local transfers of control, and lexical closures - the three building blocks that allow for construction of a condition system.
The part that comes afterwards constructs the condition system piece by piece, function after function - and only after we've constructed something from scratch, we draw parallels to relevant parts of the condition system. There's also a short tutorial for macro writing as an appendix, since we will write some macros as a part of the condition system.
Overall, this book should be able to introduce a programming into using the Common Lisp condition system. For the very basic basics, I will still want the reader to start with Practical Common Lisp, but once the reader knows the basics of syntax, evaluation rules, and basic data types in Lisp, they should be able to follow the book from the beginning.
Or, in other words, one shouldn't read a book about handling exceptional situations if one doesn't understand what are exceptional situations or why one would need to handle them.
So, I'll read your book after working through both of the others.
(Also, Common Lisp isn't a functional language. It's a multiparadigm one that often gets the label of being a functional one because Scheme and Clojure are really functionally oriented.)
Do you foresee any issues with other languages only having dynamic (i.e. THROW) non-local transfers of control rather than lexical as well (i.e. GO). I find it convenient for when I want to conditionally unwind the stack to have lexical transfers, and obviously it's been demonstrated you can implement GO with just THROW, so there's no loss of generality, but I wonder what the ergonomic impact will be.
As you mentioned, one can implement both GO and RETURN via THROW plus gensyms - see https://web.archive.org/web/20191008050814/http://home.pipel... for an implementation.
------
There have been many attempts to declare the Lisp family of languages dead, and yet it continues on in many forms. There are many explanations for this, but an obvious one is that it still contains ideas and features that aren't fully appreciated outside the Lisp community, and so it continues as both a refuge and an idea factory.
Gradually, other languages see the light and these important features migrate to other languages. For example, the Lisp community used to be unusual for standing steadfastly by automatic memory management and garbage collection when many said it couldn't be trusted to be efficient or responsive. In the modern world, however, many languages now presume that automatic memory management is normal and natural, as if this had never been a controversy. So times change.
But proper condition handling is something which other languages still have not figured out that they need. Java's try/catch and Python's try/except have indeed shown that these language appreciate the importance of representing exceptional situations as objects. However, in adopting these concepts, they have left out restarts --- a key piece of the puzzle.
When you raise an exception in Python, or throw one in Java, you are still just performing an immediate and blind transfer of control to the innermost available handler. This leaves out the rich experience that Common Lisp offers to perform actual reasoning about where to return to.
The Common Lisp condition system disconnects the ability to return to a particular place in the program from the necessity to do so, and adds the ability to "look before you leap." In other languages, if you create a possible place to return to, that is what will get used. There is no ability to say "If a certain kind of error happens, this might be a good place to return to, but I don't have a strong opinion ahead of time on whether or not it is definitely the right place."
The Common Lisp condition system separates out three different activities: describing a problem, describing a possible solution, and selecting the right solution for the right problem. In other languages, describing a possible solution is the same as selecting that solution, so the set of things you can describe is necessarily less expansive.
This matters, because in other languages such as Python or Java, by the time your program first notices a problem, it already will have "recovered" from it. The "except" or "catch" part of your "try" statement will have received control. There will have been no intervening time. To invoke the error handling process IS to transfer control. By the time any further application code is running, a stack unwind already will have happened. The dynamic context of the problem will be gone, and with it, any potential intervening options to resume operation at other points on the stack between the raising of the condition and the handling of an error. Any such opportunities to resume operation will have lost their chance to exist.
"Well, too bad", these languages would say. "If they wanted a chance, they could have handled the error." But the thing is, a lot of the business of signaling and handling conditions is about the fact that you only have partial knowledge. The more uncertain information you are forced to supply, the more your system will make bad decisions. For best results, you want to be able to defer decisions until all information is available. Simple-minded exception systems are great if you know exactly how you want to handle things ahead of time. But if you don't know, then what are you to do? Common Lisp provides much better mechanisms for navigating this uncertain space than other languages do.
So in Common Lisp you can say "I got an argument of the wrong type. Moreover, I know what I would do with an argument of the right type, I just don't happen to have one or know how to make one." Or you can say "Not only do I know what to do if I'm given an argument of the right type (even at runtime), but I even know how to store such a value so they won't hit this error over and over again." In other languages, if the program doesn't know this correctly-typed value, even if you (the user) do know it at runtime, you're simply stuck.
In Common Lisp, you can specify the restart mechanism separately from the mechanism of choosing among possible restarts. Having this ability means that an outer part of the program can make the choice, or the choice can fall through to a human user to make. Of course, the human user might get tired of answering, but in such a case, they can wrap the program with advice that will save them from the need to answer. This is a much more flexible division of responsibility than other languages offer.
After a wee bit of massaging that code to compile it on modern Lisp implemenations (the above code is CLtL1, which is an old version of Common Lisp from 1984), it still works!
It's especially not compelling that the caller passed the wrong type, the called function would like a different type and somehow a different piece of code would know both end of the situation and fix it instead of the caller or callee.
It all seems like an artificial and convoluted use case.
Software design tend to know if they'd like to abort (throw) or report (callback) statically. Conditions may be general, but they seem to me to mostly allow unnecessary open-ended complex design.
If the supposed gain is that a single system can do it all, I again don't feel it is a convincing argument. In fact, I tend to prefer a one-goal system, where different use case are easily differentiable because they are different. IOW, that throwing an exception, calling a callback, sending a signal or converting a value should look different is a plus.
This is something I've noticed: one starts with a rigid design, then adds abstractions. But one reach a point over over-abstracting where the design becomes uncomprehensible because it is so generic that it becomes meaningless.
A more contrived example would be a situation in which some piece of data (e.g. a worker's monthly timesheet report) is passed between modules of a programming system, but the receiving module, upon performing validation, discovers e.g. that the employee was working during a holiday.
Handling that situation is hard, since there are multiple ways of handling it, each of them valid in its own specific context. If the employee is in another country where that day is not a holiday, we should proceed without any other actions. If the employee has an agreement with their manager that they are having crunch time, then the system should proceed after applying overtime payment. If the employee is on a flexible time schedule, we should proceed and log this somewhere else; if the employee has no justification for that overtime, we should abort and signal an error; more examples follow.
In other words, when we signal the condition, we do not have full information about what should happen to it. We have the question "What should we do with this timesheet?" and the answer to that question is "It depends."
It is possible to model this situation by a condition type named e.g. EMPLOYEE-WORKING-DURING-HOLIDAY, and instances of that condition type being signaled inside the programming module. We create a dynamic environment where the proper handler routines for EMPLOYEE-WORKING-DURING-HOLIDAY are established, and we call the module's validator function inside that environment.
This process fully decouples the act of signaling a condition from the act of choosing whether to handle that situation and also from the act of choosing how to handle that situation.
One can (and should) document the condition type in the design specifications, and also describe functions that are allowed to signal it.
The described EMPLOYEE-WORKING-DURING-HOLIDAY is akin to a system has a slot to register a callback (or signal which can be connected to receiver, to use a Qt-like design) which can handle the situation or decide raise an exception.
Again, I understand that it might be attractive to have a single unified system to handle the different possible situations instead of separate systems. I find it hard to imagine a case where the decision to be an exception-like, callback-like or restart-like is chosen dynamically at run-time by any or all participants. And like I said, I' d be afraid that if such a case come up, it would make understanding the design harder, not simpler.
For example, as much as Qt signal/slot mechanism is powerful and flexible, I've found that when used fully, it makes understanding the code hard because it becomes impossible to know what will happen when a signal is raised because the handling is so well uncoupled.
The difference is the fact that Common Lisp has facilities that allow two things: choices of what and how to proceed and flexible non-local returns. The callback is allowed to list all recovery choices that are present in a given dynamic environment; they are established dynamically, just like handlers, and therefore can be provided fully from outside.
For instance, a handler/callback can invoke a choice named CONTINUE if it decides that absolutely nothing needs to be done and the validation is safe to proceed. The validating code doesn't need to know why exactly that choice was taken.
Or, if it notices that e.g. it needs to convert a timesheet from version 3.0 to version 4.0, it can call a conversion function on the timesheet object and invoke a choice named REVALIDATE, passing the converted timesheet as an argument. Note that the validating code doesn't need to know about any details of the conversion routine! It only needs to provide a means of restarting the validation by establishing a REVALIDATE choice.
Or, if it decides that the situation is hopeless, it can signal an error of its own and defer the responsibility of handling that situation higher up - all the way to the system debugger, if necessary. The validating code doesn't need to know about any details of why an error was signaled! That's a ton of modularity that we've just given there.
These choices are actualy named restarts in Common Lisp. I purposefully name them choices, though, as I introduce them in the book.
Not necessarily. They may return to any point on the stack that has been "announced" as a valid return point. One can compute a list of these return points, choose which one to utilize, and under which conditions.
That's what constitutes the power of restarts (and non-local exits) in Common Lisp.
I think that in a multithreaded environment you can try to hang a particular thread by forcing it into the debugger in order to preserve all stack information inside it, but I don't know if that is feasible in the long run on production systems since it opens a gate to possibly DoSing the machine.
But yea, conditions weren't designed from the start with distributed systems in mind, so you almost need a runtime that can save a whole thread to disk and recover it later. Back in the day though you could just open a window and ask the user what to do directly.
I think it depends on your OS. IIRC, on Windows, it was fairly easy to destabilize the OS by spawning threads in a loop and starving other processes on the system. I haven't used Windows in a long while though, so I cannot say for sure.
I do this with threads but my system is just a chat bot, so it doesn't really ever turn into a problem.
Picked apart I just can't see the value. You're hitting an unexpected condition then trying to handle the resultant exception intelligently, whereas you should have considered that possibility up-front and handled it non-exceptionally.
The only interesting part is for the error handler to be delegated to a human, but in a way that suggests to me that you've not done your analysis/testing/speccing properly. In a normal system I'd have something like a dead-letter endpoint for such an unrecoverable delegate-to-the-human condition. I can see the extra flexibility that would bring I admit but at what cost of gaping holes?
I feel more and more I'm missing something here.
That is the inverse of the idea. The only possibility we consider up front is that we might need to restart the process of validating the timesheet from the scratch. Why we might need to restart it, and with what data we should restart the validation, we don't really know - we let the caller specify that. If they decide that they want to make use of that restart strategy, we have successfully delegated the responsibility of making that choice and providing data for that restart elsewhere - to the caller.
We are therefore capable of decoupling the handlers and restarts from the real validation code. This means that the caller of the validation function can specify their recovery strategies at the calling site of the function or higher in the stack above it, and also that they can decide to choose to make only some of these strategies available in a context where other strategies simply do not apply.
This makes no sense. Unless you either a) alter the code or b) delegate to a human somehow[0], restarting the process is pointless - you'll get the same outcome.
> If they decide that they want to make use of that restart strategy, we have successfully delegated the responsibility of making that choice and providing data for that restart elsewhere - to the caller.
So get the caller to give that same info first time, and the exception won't happen because the exceptional condition has already been accounted for, by "provid[ing] data" (that necessary data) the first time round.
[0] which you mentioned and I did find interesting
To quote a famous person, “insanity is doing the same thing, over and over again, but expecting different results.” Hence, when we restart in this scenario, we are not restarting blindly in hope something changes; we use the restart logic to alter the input to the validator function.
* We call the validator, which in S-expressions looks like (VALIDATE-TIMESHEET TIMESHEET). This signals an error and invokes condition handlers matching that error.
* Some condition handler notices that this is a timesheet in an old format, computes an updated version of the timesheet that we will call TIMESHEET2, and invokes a restart named REVALIDATE with TIMESHEET2 passed as an argument.
* This restart sets the TIMESHEET variable to the modified value of TIMESHEET2 and transfers control to a point just before the validator is invoked.
* And so, (VALIDATE-TIMESHEET TIMESHEET) is called again, but this time TIMESHEET is in a new format, so no error is signaled and validation progresses.
All of the above can obviously be done statically, but the additional value is in decoupling the concrete means of handling the condition from the point of signaling it. The code that calls VALIDATE-TIMESHEET has no idea
------
If you consider that to be a good idea, I can write an article that gives a concrete code example of this behavior - that should be even more clear than the description that we have here. Should I?
> Some condition handler notices that this is a timesheet in an old format, computes an updated version of the timesheet that we will call TIMESHEET2, and invokes a restart named REVALIDATE with TIMESHEET2 passed as an argument.
So why not just say
if newformat(ts) do processItTheNewWay(ts)
else if oldformat(ts) do
newts = compute_updated-version-of-timesheet(ts)
processnewformatTS(newts)
you have the code for compute_updated-version-of-timesheet because you said "Some condition handler [..] computes an updated version of the timesheet that we will call TIMESHEET2" so the code must already exist.
So why expect exceptions when you can anticipate some inputs are reasonably going to cause exceptions and simply handle them upfront?
The only new thing I can think of now is that you can dynamically load a newly-written exception handler (compute_updated-version-of-timesheet, which you've rush-written because you only just found out about the old style timesheets) into a already running program to handle this unexpected condition.
The approach using restarts and the IF-based approach you've listed are fully equivalent in their functioning. I'll claim, though, that if one is capable of computing these situations up front and preparing the recovery strategies ahead of time, then we're no longer doing exception handling whatsoever. If you are able to compute all situations up front, then you don't have exceptional situations, so you don't have exceptions and therefore don't need need exception handling! If anything, one needs proper flow control; these two terms have become wrongly synonymous in some contexts. That's also how I understand your quote of "why expect exceptions".
The timesheet example is very easy to understand because we know everything about the situation: there's a new version that can be an input, an old version that also can be an input, and only one of them does not signal an error. This is why one may also decide to solve this example by using a strong type system - by defining different types (e.g. in Haskell) or classes (e.g. in Lisp) for distinct versions of the timesheet. If you can successfully use the "parse, don't validate" pattern commonly found in strictly typed strictly functional code, then one doesn't need as many exceptions because many exceptional situations become simply unrepresentable in code.
Your example assumes that one can easily couple the means of signaling a condition (conditionalizing on newFormat(ts)) and the means of handling it (calling compute-updated-version-of-timesheet(ts)) inside a singular block of code. That doesn't need to always be the case. Let's suppose that we've just run into the error and need to fix it - the situation that you described, an interactive hotfix of a running system. Fixing this properly requires modification of that piece of code that signaled an error, which requires digging into that module which we might not even have sources for, or we have the source but it's completely foreign to us and we can't afford debugging it on the spot.
We likely might need to develop a patch for that module where we now know that it has a fault and we may also need to live with it for X days before it is fixed upstream. When we have error-handling information stored in the dynamic environment and restart points sprayed across our stack, we have a situation where the error can be handled both without recompiling that faulting module and without destroying the stack - computation can continue when we've provided a proper recovery routine. We aren't getting crashes, stacks are not getting completely unwound.
So you can think of the condition system, or its combo of handlers and restarts, as an intricate system of callbacks and restart points. Every place that calls SIGNAL is a point that can execute arbitary hooks/callbacks/code; every place on the stack that accepts non-local returns is a possible place that can computation can return to in order to resume the computation. This, along with the fact that Lisp is an interactive system (a lot of Lisp's power simply disappears when it's used like Java or Rust or any other batch language), allows for a lot of possible choices of how to handle This New Situation™ that just landed us in the debugger. We can fix it interactively, and possibly store this fix in a new condition handler that we then patch on top of our running program.
Of course, this hotfix might then end up getting factored into the code properly, and what was once a toplevel condition handler that processes some data invokes some restarts becomes a proper part of the code: that situation is no longer unexpected or exceptional we are already prepared for it. (Or rather: we think that we are, and reality will validate it.)
But the condition system has served its purpose by then! It helped us keep the system alive and fix the issue on the spot.
EDIT: The main issue that makes writing examples for this troublesome is the curse of knowledge. If I write that a condition system allows to recover from a type error, it's trivial to say, "just add a typecheck for that with a conversion routine"; if I write that a condition system allows you to recover from the format mismatch, it's trivial to say "but you can just stick an ifIsOldFormat(...) in the code to solve it". These solutions are no longer about exception handling; they are modifications to the standard control flow of the program. They aren't trying to fix the situation from outside; they are trying to fix the situation from the inside. And fixing it from inside isn't always viable or even possible at all.
While control flow deals with known knowns and known unknowns, exception handling is much more about situations where we deal with unknown unknowns: something explodes, we don't know what it is, none of our programmed recovery mechanisms have taken care of that situation, and we need means of gracefully recovering.
This is achieved first and foremost by allowing human intervention: the debugger is started, the stack information is preserved, so a programmer has all the information to try and debug the situation to let the program continue execution. Some of these fixes can then be easily added to code in by means of defining new programmatic condition handlers (hooks) and restarts (choices) in code.
(EDIT2: I wholeheartedly apologize for the wall of text. I'll just add it to the book instead - thank you for the good questions.)
> I'll claim, though, that if one is capable of computing these situations up front and preparing the recovery strategies ahead of time, then we're no longer doing exception handling whatsoever. If you are able to compute all situations up front, then you don't have exceptional situations, so you don't have exceptions and therefore don't need need exception handling!
Exactly! This is a cultural difference; in my world you should only use exceptions for things that should never happen, but I suppose 'never' is a matter of expectation and therefore code that follows that expectation.
> This is why one may also decide to solve this example by using a strong type system
Agreed, and to me this is the 'correct' solution. But that may be a counsel of perfection (and an overbearing cultural 'arrogance' on my part)
> That doesn't need to always be the case. Let's suppose that we've just run into the error and need to fix it - the situation that you described, an interactive hotfix of a running system.
This is both cultural and technical difference that I've been struggling with. This makes it clear what you're getting at.
> If you can successfully use the "parse, don't validate" pattern[...]
Not heard of this. You've given me some reading to do, thanks. For those unfamiliar, https://html.duckduckgo.com/html?q=parse%2C%20don%27t%20vali...
> We likely might need to develop a patch for that module where we now know that it has a fault and we may also need to live with it for X days before it is fixed upstream. When we have error-handling information stored in the dynamic environment and restart points sprayed across our stack, we have a situation where the error can be handled both without recompiling that faulting module and without destroying the stack - computation can continue when we've provided a proper recovery routine.
That is a compelling use case.
> We aren't getting crashes, stacks are not getting completely unwound.
I came across Lisper's enthusiasm for such exception handling ability (based on having the stack not-unwound) when I was trying to learn Dylan, a long time ago. I'm starting to see what they were getting at. That doesn't mean I'm comfortable with it, but I do understand it rather better now.
> along with the fact that Lisp is an interactive system (a lot of Lisp's power simply disappears when it's used like Java or Rust or any other batch language) [...] We can fix it interactively, and possibly store this fix in a new condition handler that we then patch on top of our running program [...] But the condition system has served its purpose by then! It helped us keep the system alive and fix the issue on the spot [...] These solutions are no longer about exception handling; they are modifications to the standard control flow of the program. They aren't trying to fix the situation from outside; they are trying to fix the situation from the inside. And fixing it from inside isn't always viable or even possible at all [...] This is achieved first and foremost by allowing human intervention: the debugger is started, the stack information is preserved, so a programmer has all the information to try and debug the situation to let the program continue execution
OK, these explicit statements are very important and revealing.
> ...apologize for the wall of text
No need, it's an excellent answer, it's just a perspective that does not come from 'my side of the tracks'.
Glad I persisted, I feel I've learnt something important today. Good luck with the book!
How should I credit you in the book's Hall of Fame?
Only if it's presented as a "motivating example", rather than "illustration".
This is the TXR Lisp interactive listener of TXR 240.
Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet.
1> (defun filter-using-exceptions(list)
(build ;; build macro: implicit, procedural list building
(each ((item list))
(catch (throw 'keep-it? item)
(yes () (add item))
(no ())))))
filter-using-exceptions
2> (handle (filter-using-exceptions (range 1 10))
(keep-it? (item) (if (evenp item) (throw 'yes) (throw 'no))))
(2 4 6 8 10)
3> (defun filter-using-exceptions (list)
(build
(each ((item list))
(catch (throw 'keep-it? item)
(yes () (add item))
(no ())
(substitute (this) (add this))))))
filter-using-exceptions
4> (handle (filter-using-exceptions (range 1 10))
(keep-it? (item)
(cond
((< item 5) (throw 'yes))
((= item 5) (throw 'substitute 42))
(t (throw 'no)))))
(1 2 3 4 42)
In TXR, I simplified the concept by eliminating the duality of conditions and restarts. There are only exceptions. catch frames take an exception with unwinding. handle frames take an exception without unwinding (they can potentially decline it after looking at it). In the above example, keep-it? is an exception and so are yes, no and substitute.
So, why would you do the above instead of just, say:
5> (mappend (lambda (item)
(cond
((< item 5) (list item))
((= item 5) (list 42))
(t nil)))
(range 1 10))
(1 2 3 4 42)
In the exception example, that is happening also: a function is required from the caller to complete the behavior. But that function isn't passed as a parameter; it is dynamically bound as a handler which is labeled by an exception symbol. The function doesn't simply return a value, but chooses from a menu of restart points.
That points to a whole different set of ways of organizing the code, and options for extending and maintaining it.
Firstly, for pretty much what the parent describes. Essentially, to help you continue debugging in the presence of incompatible caller/callees, typos etc. Even with very disciplined developers, these are pretty common in image based, dynamic environments (Smalltalk has similar issues in my experience). Re-starting from scratch would be exceptionally painful so a "patch and continue" approach is very useful. I fully concur that this isn't a compelling use case even in weakly typed languages.
Secondly, for exception throwing and catching of similar ilk to Java/C# etc. In other words, in cases where a less general condition system would have worked as well.
Thirdly, where there are 2 or 3 reasonable options to recover from an error condition, generated in library code, and these are coded with the error. The policy regarding how to handle the error is then made at the application level. You could do something similar without the CL condition system, say by passing flags around (or using globals) and switching on them, but this is super cumbersome/ugly and I don't think I've ever seen it used much.
But honestly, there were maybe 2 or 3 cases where the third use wasn't better coded as "try this one thing and, if that doesn't work either, bail".
So my assessment is that it's not crossed over to other languages mostly because of its limited usefulness in those environments. I contrast this with garbage collection and compile time coding which have more general utility and have successfully crossed over.
Multiple dispatch hasn't crossed over much either (the obvious exception being Julia via Dylan). Now, I did happen to find that very useful so I'm a little more surprised at that than I am at the generalised condition system.
First, I describe how the condition system can be used to work with conditions that are not errors and therefore do not require to be handled; Peter Seibel calls SIGNAL a "primitive" function, therefore completely skipping the scenario in which a condition may not be handled. And, by that, he skips non-error non-warning conditions altogether!
Second, the PCL chapter does not describe the internals of a condition system, how the handlers and restarts are constructed. My book describes that in detail twice: once, building a sample condition-like system piecewise, and then again, describing the implementation of a complete, ANSI-compliant condition system.
http://jacek.zlydach.pl/blog/2019-07-24-algebraic-effects-yo...
The first part uses an example that you'd probably find not all that convincing (though it's a reworded example from Dan Abramov's post). The second one is not about error handling at all - but instead, shows how you can use non-error conditions to bolt on a pseudo-UI on top of an operation that signals appropriately, using both restarts (to abort the operation) and signal handlers (as sort of observer pattern).
I'd say the magic of CL's condition system is in restarts, and the ability to choose them programmatically. It's a powerful tool in API design, that allows you to cheaply expose out-of-band interface for monitoring and error recovery (and I mean actual recovery).
Suppose you're writing a module that extract data from a bunch of files (possibly accessed over network). In the high-level view, you have two layers there: one that loops over files, and the other that goes over data in a given file and extracts values of interest.
Writing this in Common Lisp, you could define a restart in the loop layer, allowing to retry a download, or substitute a different file URL. Then, in the file processing layer, you could define a restart around reading an invalid value, allowing to skip it or substitute it. These error conditions and restarts are now part of your module API. The users of that module could then choose different strategies for error recovery. In one case, if file access fails, you need to abort everything. In another, you need to skip it. In yet another, the caller knows of a backup data file, and can pass that information to the restart. Similarly, invalid values could be replaced by an appropriate default value, again provided by the caller in the context.
I'm trying to read in some Lisp code, from a wide variety of sources, as grist for a mutational fuzz tester. I don't want to have to define all the packages used in this code. So, when I come across FOO::BAR I want to be able to gracefully handle this rather than causing a reader error if the FOO package is undefined.
Restarts let me do this. In SBCL, there's a restart that let's the handler use a different package in this case. In my case, I just stuck the symbol in some other default package.
One could also imagine this being used as a load-on-demand system. The handler could look up FOO in some list of packages provided by various libraries and automatically load the right library, then return control to the reader to try now that the package was defined.
The reader itself doesn't have to know about any of these various ways of dealing with reader errors; it just has to provide the restarts so the various handlers can tell it how to proceed.
The slots and/or accessor methods via which one can access the condition objects can (and should!) be documented as a part of the contract of any piece of code that is expected to signal conditions.
Resuming execution is possible since the original stack is not destroyed by unwinding it. The handling code can simply perform a transfer of control to a point on that stack established earlier and continue from there.
The behavior is also discoverable interactively. You play with widget-frob at the REPL, and hit a widget-error condition. This recurses into a debug prompt where you are told about the available restarts, which have docstring-like descriptions.
Sometimes a single operation has multiple ways of failing, each with their own ideal recovery / retry plan.
You want to catch the error, examine context (app state, error type and metadata at crash point) and determine what the best next step is.
This example linked below is from a web crawler service I run.
When an error occurs while logging in (multiple page loads and sometimes a redirect), there is a single handler to determine if retrying the login, crashing or alerting a human is the best course of action.
https://twitter.com/akamaozu/status/1193910918641532934
Ended up with a way to write code that recovers from errors in ways that I might not immediately understand but demonstrably works.
It could also choose to invoke a restart. Restarts are basically code to handle a problem, but they don't catch any conditions. Instead, they can be invoked, usually from a condition handler. So you would have an error being signaled at the lowest level of your code, various recovery strategies defined at appropriate intermediate levels of code, and at the highest level of code you could have a condition handler defined which picks whichever restart is best for the high level situation. So a library author can provide many different restarts, signal errors very deep in the library, and then let the library user choose the recovery strategy that fits their requirements.
If no matching handler is found, typically the debugger would be invoked, and when running with a suitable IDE like Emacs with Slime or Sly, a window with a stack trace will pop up. Here, if there were any available, restarts can be invoked directly by the programmer while they are debugging
Finally, the condition system can be used to signal non-erroneous conditions, that can optionally be handled.
If you are just going to use `TAGBODY` and `GO` to implement this ... in a goto based languages like C I don't think you would be needing a condition system. In C a nested function can also jump into a parents goto label. No one probably uses it, but just saying!
Also how similar is this to promises in javascript ?
The book contains an implementation of dynamic variables in C, though, which uses a GCC cleanup extension; the example code contains the contributed code examples for that showing various means of implementing dynavars in C.
Promises are asynchronous while signaling and condition handling is fully synchronous; I don't know what kind of parallel one can draw here. If anything, a promise's .catch(...) method may act like a CL HANDLER-CASE; the promise becomes resolved, just a different code block is executed in case of a failure.
What I meant with the promises was, If you could pass three closures ... like success, error, restart could you get some kinda condition system ?
And that also only handles the case of "help I'm ded get me out of here". What about signals that do not expect to be handled, and instead are used to transfer information higher up the stack by invoking a handler function specified in the dynamic environment? That's a valid use of the condition system, as outlined in the book.
Works for perl because while we don't (yet, somebody's working on one) have an unwind-protect like primitive, perl's refcounting system provides timely destruction so you can use a scope guard object stuffed into a lexical on the stack whose DESTROY method does whatever unwind cleanup you need.
Ironically, the main reason I'm not using this so much at the moment is that it isn't compatible with the suspend/resume code around promises provided by Future::AsyncAwait and I'm heavily using that in my current code, but at the point where I need both I'll probably attempt to nerd snipe one of the relevant authors into helping me figure it out.
(EDIT: Aaaaactually, I think I might already know how to make them work together, naturally an idea popped into my head just after I hit the post button ... using Syntax::Keyword::Dynamically and capturing a relevant future higher up the chain of calls should allow me to return-to-there cleanly, then I "just" need to cancel the intermediate futures to simulate unwinding ... but I'll have to try it to find out)
Thank you for mentioning Worlogog, I'll include that in the book. How should I credit you?
"Matt S. Trout (mst)" is good - I'm probably better known by my IRC nick than my full name in geek circles ;)
I imagine that is the reality of everyone who tries to implement a condition system in a language where the default behavior is to immediately unwind the stack by throwing some sort of an exception.
If I were to want to continue discussing this, do you happen to kick around anywhere on IRC that would be suitable?
Well, once upon a time it used to be setjmp/longjmp; currently it's based on a re-implementation of a mechanism that is almost setjmp/longjmp in assembly language.
To use setjmp/longjmp for implementing sophisticated exception handling, you have to maintain your own unwind chain on the stack and unwind that.
Before invoking longjmp, you have to walk your own frames that are chained through the stack, and do all the clean-up yourself, up to that frame that holds the jmp_buf where you want to jump.
TXR correctly maintains the chain connectivity even under delimited continuation support, which works by copying sections of the C stack to and from a heap object.
When a delimited continuation is restored (involving copying it out of a heap into a new location on the stack), the frame linkage in the restored continuation is fixed up and hooked up. The continuation can throw an exception and unwind out through the caller that invoked it.
Here is the implementation of unwind-protect operator in the interpreter:
static val op_unwind_protect(val form, val env)
{
val prot_form = second(form);
val cleanup_forms = rest(rest(form));
val result = nil;
uw_simple_catch_begin;
result = eval(prot_form, env, prot_form);
uw_unwind {
eval_progn(cleanup_forms, env, cleanup_forms);
}
uw_catch_end;
return result;
}
In the virtual machine, there is a uwprot instruction instead:
1> (disassemble (compile-toplevel '(unwind-protect (foo) (bar))))
** warning: (expr-1:1) unbound function foo
** warning: (expr-1:1) unbound function bar
data:
syms:
0: foo
1: bar
code:
0: 58000004 uwprot 4
1: 24000002 gcall t2 0
2: 00000000
3: 10000000 end nil
4: 24000003 gcall t3 1
5: 00000001
6: 10000000 end nil
7: 10000002 end t2
instruction count:
6
#<sys:vm-desc: 9432d90>
The function that the vm interpreter dispatches for uwprot is simplicity itself:
NOINLINE static void vm_uwprot(struct vm *vm, vm_word_t insn)
{
int saved_lev = vm->lev;
unsigned cleanup_ip = vm_insn_bigop(insn);
uw_simple_catch_begin;
vm_execute(vm);
uw_unwind {
vm->lev = saved_lev;
vm->ip = cleanup_ip;
vm_execute(vm);
}
uw_catch_end;
}
It positively cannot, because,
1. There is no such thing as a nested function in ISO C. It's a GCC extension.
2. Let's try it the GCC extension:
#include <stdio.h>
int main(void)
{
void foo()
{
goto out;
}
foo();
puts("skipped");
out:
return 0;
}
nestedgoto.c: In function ‘foo’:
nestedgoto.c:7:6: error: label ‘out’ used but not defined
goto out;
^~~~
#include <stdio.h>
int main(void)
{
void foo(void *target)
{
goto *target;
}
foo(&&out);
puts("skipped");
out:
return 0;
}
$ ./nestedgoto
*** stack smashing detected ***: <unknown> terminated
Aborted (core dumped)
[8]> (tagbody
(funcall (lambda () (go out)))
(print 'skipped)
out
(print 'out))
OUT
NIL
I look forward to reading your book when it is released!
I have been using Common Lisp since about 1982 but to be honest, I have a lot to learn about its condition system.
Except lacking deep more coverage applications, I think Practical Common Lisp covers all of that.
Which isn't a vote against a book focussed on the subject, though.
It does not describe any applications or use cases of the condition system that do not strictly serve exception handling, and it does not show the internal construction of the condition system, especially from the approach of starting with nothing but most basic language tools which are dynamic variables, means of transferring control, closures, functions, classes, and variables.
However think of all the things that can go wrong with your environment that don't involve a bug in your program. A network connection goes down, the disk gets full, &c. All of those can be handled without unwinding the stack to the point where the code for handling them was defined.
For a really trivial example, consider a performance watchdog. If you want to completely abort the attempt and log a call-stack, you can do that it just about any modern, dynamic language because the exception object includes the call-stack. But what if you want to log the call-stack and continue for the first N times the watchdog rolls-over? This is, of course, doable in other languages with passing callbacks and such, but in Lisp this can be done by signalling a condition in the exact same manner that an exceptional condition would be signaled.
I'm excited about this book, because that particular feature is underutilized in Lisp (by myself included), probably because it's one of the last dynamic things that Lisp does that more well-known dynamic languages do not, so programmers coming from other languages just aren't going to use it.
Nitpick: the only thing that cannot be handled by winding the stack is when you run out of stack to wind.
But, luckily, if your program overflows the stack, you usually have bigger issues to deal with and/or you usually have all the stack information you need to diagnose the problem.
It might not be easy to tell Clojure or the JVM to not be trigger happy with throwing exceptions and therefore destroying the stack. Perhaps one will find a way through!
The credit goes to the author of that library, though. :)
All the best for your book!
With traditional exceptions, the stack unwind all the way to the "catch" block which has full responsibility of handling the exception. With CL conditions, you can implement multiple handlers which do not catch the exception, and then a handler way up in the stack can decide which of the low-level handlers to run.
It sounds bonkers, but when you've tried it you'll see that "normal" exceptions are a very crippled version of it.
I'd say that this is the main difference that sets the CL condition system apart from mainstream programming languages. In these other languages, when an error happens, the stack is automatically unwound; in Lisp, when an error happens, the stack is automatically wound further, and the code executed in such way has the full choice of whether to unwind the stack and where exactly to unwind it to; this information is provided in the dynamic environment in which a given piece of code is running.
When you divide by zero in Java, then there is nothing set in stone that prevents the language from winding the stack further and executing some code that will analyze the dynamic environment in which the error happened, calling all error handlers found in the dynamic environment, and only then - if no handler transferred control outside the error side - either entering an interactive debugger of some sorts or giving up and crashing.
But instead Java goes another way and immediately destroys the stack by throwing an exception. That's a design choice made by Java creators. Lisp's homoiconicity has nothing to do with this.
You can also analyze the Python implementation of the condition system posted elsewhere in the comments - that's an example of a condition system in a non-homoiconic language.
The image-based Lisp is just very very different when compared to mainstream batch languages like C, Rust, or Java.
But that only wants to make me learn it properly more.
In languages with "exceptions", when an exception gets thrown, you have a chance to catch it, but at that point the stack has already been unwound and there is no way to resume execution at the point where exception was thrown.
In CL, you can let someone else (the caller) make the decision on what to do in case of an exception, providing options. For example, if a value is a NaN, the caller could request to continue the computation using the caller-supplied value instead.
In practical terms, this system, while very impressive, it isn't widely understood or used in CL.
It's one of the very few things I miss after I moved from CL to Clojure. Rich Hickey wrote that he did consider it, but it was a lot of complexity for fairly little gain (all platforms where Clojure is hosted support thrown exceptions only), which I would agree with.
I agree and wholeheartedly hope that my book mends that issue, even if just partially.
However, that doesn't need to be the case; it is still possible to e.g. do the industry standard, which is to log all available information about the request and program state to some sort of log file or crash dumps, return a HTTP 500, and start waiting for another request. The possibility of interactive resolution of exceptional situations is a possibility, not a requirement.
It's also possible to block and invoke the debugger only in some cases, e.g. when some global debug variable is set to true. In all other cases, the system can behave automatically.
>log all available information about the request and program state to some sort of log file
I've only dabbled with some Schemes, but given Lisp's homoiconic nature, I'm guessing it would also be possible to generate and persist a unit test based on the failing call too, right?
As long as you can persist all the state (which is easy if you are programming in a functional manner), then you will know exactly what sort of input caused a crash in your program. And when you have the data, you can write it in a manner that is later readable by the Lisp reaer.
It also means you don't have to write blinks of try catch nested in while loop just to be able to retry the operation...
The roots of condition system evolved from two related hacker environments: the ITS system at the MIT AI lab and the lisp machines (both CADR and its descendants and the PARC D-machines) as well as influence for the Smalltalk folks.
In ITS what is called the “shell” these days was the debugger. If your program signaled an error you were immediately in the debugger and could debug, just exit, or Force a core dump. This was the routine way to interact, even for non-programmers. The lispm did the same. Debugging was a lightweight operation.
So it was natural in the 80s to develop a system that formalized this mode of operation. Because an exceptional situation by definition requires greater knowledge than is available in situ. When you catch an error in java or can++ it’s too late to look at the state of the object that caused the error without standing on your head and possibly redesigning the try block say to allocate something on the heap that can be passed to the error (which means no more RAII etc etc).
In addition, try/catch typically puts the hand lin too close to the site of the problem for you to be useful; if not, by the time you get the exception there’s not much you can do but terminate the calculation politely.
This seems like two core issues with understanding the condition system. First, the currently popular exception systems always destroy the stack by unwinding it, allowing the programmer to debug only dead remains instead of a living program; second, debuggers are bolted on top of programs instead of being their internal parts, which gives many people an impression that using them as an integral part of the whole programming lifecycle is inconvenient and unnatural.
The condition system in Common Lisp solves both of these issues, which IMO, paradoxically, contributes to it not being well understood outside Lisp circles.
I tried to implement this in Ruby with a gem about 6 years ago and got really close. If anyone wants to know more about this, but is more fluent in Ruby, you might find this a good jumping-off point. There’s a good explanation in the README.
https://github.com/michaeljbishop/mulligan
Two problems are:
1. some code may fail multiple times (with non-async code, once an exception is caught the program can’t throw another one from the same dynamic extent of the try block; similarly once a restart is invoked and control is transferred, a condition won’t be signalled from the same dynamic extent where the restart was bound (for the purpose of this clause, we consider a retry restart to mint a new dynamic extent to bind the restart in again after retrying))
2. Some computation may happen after you don’t care about its result because a parallel dependency has failed. E.g. suppose you have some rate limiter where you pass an async function, get a promise for it’s result, and the rate limited will queue your function to be called soon but with the condition that it will only allow say 5 calls to be in flight. Now you have some resource-downloading function which puts many separate download requests into the ratelimiter in parallel, and combines the results in such a way that the whole download fails if any request fails. But now if an early request fails, your function will continue trying to download all the other resources even though it doesn’t care about the result now that it has failed. You could imagine having a way to cancel promises and maybe you would be clever and write really solid programs and never have issues but to me this seems to be inviting very difficult to debug problems (just like expecting any thread to be killed at any random point (or at least a bunch of different points) would make correct multithreaded programming harder.
Perhaps a more succinct description of problem 2 is that unwinding the “stack” isn’t well-defined with an async programming model because “stack” doesn’t really mean the call stack from the scheduler you whatever bit of async computation you happen to be doing, but rather the combined dynamic scope from everything that started the computation and is currently awaiting its completion.
A related and, I think, very useful thing to have is a way to represent computations which may signal conditions or be restarted. I think CL doesn’t have a great story for this (threading isn’t really a part of the language though). An example use case would be some program which has to process some stream of events. If it gets stuck on one or some, you may want to continue processing the rest of the events while deciding what restart to invoke for the problematic ones. Perhaps an effect system could help with this.
This topic is still waiting to be explored, either with an extension of Common Lisp or a completely different language that permits asynchronous programming with an adaptation of the condition system to this programming paradigm.
There is no link to buy the book. Sigh.
When your book is in print and available for sale I hope you post again here as a reminder.
Thank you for the kind words; I hope that the book suits you well once it's out.