Lisp Operating System (2015) | Hacker News Reader

Lisp Operating System (2015)

metamodular.com

173 pointsantipropaganda61 comments

Summary:

- We don't need separate processes if the language we use is inherently safe (e.g., modern Common Lisp).

- Without process isolation, it is possible to share large, complicated and possibly mutable data structures (graphs, arrays, user-defined objects) system-wide.

- Once it is possible to cheaply share and communicate arbitrary data structures, it is pointless to maintain designated 'file system'.

- Not having a file system raises the question of what data should be persistent and what shouldn't. But on a modern computer, it is feasible to just treat all data as persistent (maybe excluding the youngest GC generation).

The best news is that the author is actually working hard to implement this operating system. The first part - the Lisp implementation - is already in a pretty good shape and could be finished within the next two years: https://github.com/robert-strandh/SICL

Should mention that Mezzano OS satisfies most of these design points and already runs.

https://github.com/froggey/Mezzano

I wanted to also mention Mezzano - really easy to play with using VirtualBox.

A bit off topic, but when I miss the environment of the Xerox 1108 Lisp Machine that I had from 1982 to about 1987, I find the closest thing today that offers a similar experience is Pharo Smalltalk.

I've had some user interface ideas I've wanted to explore floating around in my head for more than a decade. I' m now in a position to start exploring these ideas and after an extensive survey I decided to use Squeak. The deeper I get into this system the more appreciation I gain for the entire model.

Be sure to check out Cuis and Pharo as well.

I like Pharo but Morphic seems to be much more front and center in Squeak. And becuase I'm focusing on are new UI paradigms the Morphic interaction is really what I'm concerned with.

Dan Ingalls wrote similarly in his Design Principles Behind Smalltalk [1], noting that "An operating system is a collection of things that don't fit into a language. There shouldn't be one."

[1] https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk....

That makes me reconsider the Plan9 OS, where everything is a file system.

Then, nothing fits in the language (C)?

Well, the extremely generic interface of Plan9 (plaintext files) can make you use any language. It is the extreme difference with a OS which is in itself one language.

Making everything a file is one way to give all components a universal interface. It's not the only possible or even the best way. The problem is that today there are no universal interfaces at all, so even a file-based one seems immensely powerful by comparison.

yes, files in plan 9 can be viewed as a way to "make objects look like files."

http://doc.cat-v.org/plan_9/4th_edition/papers/names

note that the filesystem interface isn't totally durable across all operations: a syscall is still necessary to create processes, for instance. cp /proc/... doesn't do what one might think it does.

http://doc.cat-v.org/plan_9/4th_edition/papers/9

i appreciate that the original article similarly acknowledges that lisp is an "all the things (except for some things)" solution in the Single address space section. synchronization libraries (and perhaps other libs as well?) in C-based OSs need to drop into assembly to get at test-and-set kinds of operations. addressing these special cases one-by-one is something that'll need to be ramified in source in order to get any real insight, although (perhaps just from the inertia of familiarizing with the original bell labs warp) it's difficult to imagine the kernel as a "all the things, seriously everything" abstraction away from the machine.

----

overall, the original article and discussion here was a bright spot in my day. it gives me hope that there's some actual thought and discussion about how to evolve operating systems intended for commodity hardware, not just generalized, "yea, that's something we could do," one-offs. i'm not as familiar with the xerox heritage mentioned elsewhere in the comments, and the view that, in some sense, linguistic abstraction might cover OSs is something to think about.

i've heard the smalltalkers catch some flak about not actually writing an os because there was (apparently) the equivalent of exec(), no fork()

http://bitsavers.org/pdf/xerox/alto/memos_1975/Alto_Operatin...

this criticism of the alto OS could be due to cognitive bias as much as anything else. the concept of processes is by now deeply ingrained in the way we conceptualize operating systems. so i have to ask: is leading with eschewing processes a clean-sheet rethink, informed by history's mistakes as well as its successes, or is limiting the number of OS primitives at the expense of less-rich interfaces actually a desirable tradeoff?

like, although everything i've heard about multics in particular seems well thought-out, thomson and ritchie were doing something substantially different by opting for bytestreams as much of the interface rather than making strict decisions about arities and so on. every rule system makes is bound to be a rule a user will eventually want to break. i suppose the operating system's main job is to safely lift off of the hardware, not to impose further unnecessary artifice (of which hierarchical file systems AKA namespaces could be viewed as one) on the user. adding further icing onto this core objective is so much the better, so long as it's possible to scrape away and redecorate.

> i've heard the smalltalkers catch some flak about not actually writing an os because there was (apparently) the equivalent of exec(), no fork()

In the Smalltalk world, processes are just instances of the class `Process` and `Processor` is a singleton that manages all processes. You can create new processes several different ways, the most common being to send the message `fork` to a block closure.

Because everything is so late bound in a Smalltalk system, and because the objects are running and executing live all the time, it comes to resemble an OS in a lot of ways. I think the key lesson here is not just about which language is better or good for X or Y, but about how holistic computing environments are more important than languages alone. I go on a lot about Hypercard and how that was a real missed opportunity, as well as being an excellent environment. But it's Hypertalk language was only a part of that environment and cannot really be evaluated on its own, for example.

There is a large interview with Chuck Moore of Forth fame on O'Reilly Safari Online.

Much of the interview focuses on things like how an OS is uneccesarry and shouldn't exist. Granted, he has drastically different needs than the average user.

> Not having a file system raises the question of what data should be persistent and what shouldn't. But on a modern computer, it is feasible to just treat all data as persistent (maybe excluding the youngest GC generation).

I'd be concerned about the UX issue of ephemeral vs. permanent changes here. I doubt that files and save/revert operations are the best we can do, but I think there's a lot of value in the fact that some pattern for it exists that's common to most applications. Perhaps this has already been tackled in some system that doesn't natively use the concept of disk files. It could be as simple as having a pattern to assign names to persistent copies of objects, or something more sophisticated like assigning names to points in an undo tree that are then transparently converted into self-contained objects when the tree is pruned.

If you squint a bit, the web is a bit like a giant multi-user operating system. We certainly save a lot of data in it, and expect it to persist. But we don't generally do that using a file metaphor, and i don't hear people crying out for one. My comments on HN don't look like files to me. Nor do my tasting notes on Untappd, my shitposts on various Slacks, my projects on GitHub (which contain files, but aren't files themselves), etc. So we already have an existence proof for an operating system without files.

Now, whether the web would be better if everything was more file-like is an open question. That would be very interesting, and probably closer to the Nelsonian ideal.

Except, everything you mentioned are still one or multiple files sitting on a remote server. That it isn't necessarily presented as such through a particular web app's interface isn't relevant... under the hood, it's still just files on an operating system designed around that concept. Even data in a database is also data in a database file.

I mean, there's a reason URLs look like directory paths... that's what they were originally meant to be, remote paths to files.

> Except, everything you mentioned are still one or multiple files sitting on a remote server.

Yes, in the same way that this LISP OS's storage is still extents of bytes on a disk indexed by a lookup structure, ie files.

> That it isn't necessarily presented as such through a particular web app's interface isn't relevant.

Not only is it relevant, it is the entire point of the comment.

I'm specifically not suggesting that systems necessarily need a file metaphor. I'm concerned about what happens when every app has its own idiosyncratic UX pattern for saving/committing your work. I know people who have lost work in web apps (e.g. issue tracking systems) because it's not obvious when their changes are actually committed as opposed to sitting in memory in their browser, and people who have been bitten by weird interactions between auto-save and undo in IDEs.

> We don't need separate processes if the language we use is inherently safe (e.g., modern Common Lisp).

I note the year of the article, which was before Spectre and friends showed that this is not a reasonable assumption these days[1].

So does this mean LispOS is doomed?

[1]: https://arxiv.org/abs/1902.05178

If you have a single address space you don't need Spectre to be able to discover any aspect of the running system.

How would you then circumvent the protection?

Only if you're given raw memory access, which you are not in Lisp.

You were given raw memory access on historical Lisp Machines.

Without raw memory access, good luck supporting anything than basic hardware. Even some USB features requires it.

It's entirely possible to write a system where application code only gets sanitized handles and OS code inside the implementation gets raw memory pointers at least some of the time. All running in one address space doesn't mean all running in one security context, necessarily.

Really, address spaces are just a hardware implementation of a more general concept: Namespace-based security. Application code wouldn't even be able to know the names of privileged objects, and if someone told that code the right names, it wouldn't be able to use them, because resolution of names to things is, itself, privileged.

In a simple example, assuming a Common Lisp-like system: Everything which handles raw pointers is in the SYSTEM package. Application code can't inspect the SYSTEM package, and, even if you told an application that the function SYSTEM:WRITE-DATA-TO-DISK was a thing, it couldn't call that function because the real evaluation code, which can see into the SYSTEM namespace, knows not to let application code call anything in SYSTEM; only the functions in SYSTEM and SYSCALL can do that.

"Lisp" is a family of languages; what you're given in a family of languages is ... languages. What we find under a language is one or more implementations, and what you're given in some implementations is various forms of raw memory access.

TXR Lisp: let's gain access to malloc, allocate 42 bytes and then pretend that this block contains 100 "uint32" words (400 bytes):

  This is the TXR Lisp interactive listener of TXR 215.
  Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet.
  1> (with-dyn-lib nil (deffi malloc "malloc" cptr (size-t)))
  #:lib-0010
  2> (malloc 42)
  #<cptr: 9158a18>
  3> (carray-cptr *2 (ffi uint32) 100)
  #<carray 100 #<ffi-type uint32>>
  4> (vec-carray *3)
  #(8381552 8381552 0 0 152124648 152288384 152288632 152288800 152288952
    152289168 152289336 121 0 60 10 0 16 0 0 0 128 16 0 0 0 152292000
    0 0 0 152126920 151513152 152292584 152127032 152292680 152292896
    152293096 152293240 152293368 0 0 152293752 73 35 60 99 112 116
    114 58 32 57 49 53 56 97 49 56 62 0 73 40 118 101 99 45 99 97
    114 114 97 121 32 42 51 41 0 72 57 40 109 97 108 108 111 99 32
    52 50 41 0 152304256 121 0 60 10 0 3 0 0 0)

TXR Lisp has good sandboxing for applications. The data structure which maps symbol package names to packages is itself a variable that can be overridden. So we can create a sandbox in which the sys package doesn't exist, and the usr package prefix refers to an altered one that has been scrubbed to a safe subset of symbols.

http://nongnu.org/txr/txr-manpage.html#N-00E20381

The sys:bits function is available now:

  1> (sys:bits (cons 1 2))
  3076696060

  2> *package-alist*
  (("pub" . #<package: pub>) ("usr" . #<package: usr>) ("keyword" . #<package: keyword>)
   ("sys" . #<package: sys>))

Remove the "sys" package:

  3> [remqual "sys" *package-alist* car]
  (("pub" . #<package: pub>) ("usr" . #<package: usr>) ("keyword" . #<package: keyword>))
  4> (set *package-alist* *3)
  (("pub" . #<package: pub>) ("usr" . #<package: usr>) ("keyword" . #<package: keyword>))

The symbol sys:bits is not recognized any more:

  5> (sys:bits (cons 1 2))
  expr-5:1: sys:bits: package sys not found
  ** syntax error

Because package-alist is a special variable, it's possible to dynamically override with let rather than globally clobber it in the above manner. The dynamic environment can then be a sandbox.

Only doomed on broken hardware. I think that mitigating Spectre and similar attacks via software is a losing battle. Eventually hardware will have to be fixed to properly isolate microarchitecture state during speculative execution.

Also technically we still don't need separate processes to mitigate spectre, we just need to make sure that microarchitecture state is flushed when changing security domains. This is often accomplished with processes, but it could also just be accomplished by marking system methods in a LispOS as "priviledged" and decorating them to flush the microarchitecture state when called.

"inherently safe" - for which definition of "safe"?

The problem with the safety requirement, is that it's not just a language thing. It's also an architecture thing.

We created sandboxes for a reason.

For the incremental development the author describes, you can do that atop any GNU/Linux host platform on which your CL implementation runs.

To play with a bootable user experience, before you've replaced the host kernel and device support, you can strip down something like Debian Live. I previously did this for a Racket-based living room appliance project. A simpler example, without Racketisms, that uses only 2 shell scripts (you only need 1; the other is to play partition table tricks), is my earlier Debian Live variant: https://www.neilvandyke.org/lildeb/

I'm also a fan of a variation on the author's approach: have your better/different language get a foothold on a working GNU/Linux system, alongside traditional apps, in a system the user could use as their daily driver, and incrementally replace all of userland. This was my plan for PostmarketOS (though I've suspended my open source work on "non-employable" things): https://www.neilvandyke.org/postmarketos/

The author's approach also seems valid, and worth playing with.

> I'm also a fan of a variation on the author's approach: have your better/different language get a foothold on a working GNU/Linux system, alongside traditional apps, in a system the user could use as their daily driver, and incrementally replace all of userland.

That's incidentally what Emacs is to many of its users (myself included).

I used to do this with Emacs, too (VM, Gnus, w3m, and more besides all the programming tools), and some of my Emacs packages are in support of that: https://www.neilvandyke.org/emacs/

For a (mostly) single host process, without preemptive multitasking/threading, Emacs as an operating system works surprisingly well.

I’m all for thinking outside the box but many of the complaints described in that are blamed on UNIX when in fact they were also adopted by a multitude of other non-UNIX platforms and reason they became common was because it simply made more sense for general computing.

For example their “file system” (for want a better description) approach was tried on an IBM mainframe and it proved more cumbersome to maintain than the typical file system approach we have currently.

Also when talking about address spaces he acknowledges traditional memory management exists for security reasons then says that’s not needed in his OS without actually proving how his method either prevents the same attacks nor new ones that effectively result in the same kind of vulnerability (I could think of a few very easy attacks against his OS design).

Similarly the way he combines physical storage and system memory into one device completely overlooks technical limitations of the hardware, the risk of data loss if you don’t manage things correctly, the vastly different storage capacities between the two, nor the fact that basically every OS does this to some extent already via caching.

While it’s nice to go back to the drawing board and rethink certain idioms, this felt more like a half baked rant at UNIX than an objective look at the past and current OS designs.

That was my take away as well. Some interesting ideas - most of which aren't new - but the ideology felt very biased. To a fault even. I would still love to see a working version of this. Or even just a more traditional general computing OS but one which used FP (not necessarily LISP) as its primitive.

This article I read yesterday has one interesting take on what an OS designed on modern programming principles could look like. It has a loose treatment of technical FP terms, but the insights behind it are at least thought-provoking.

https://medium.com/@reinman/monoids-to-groupoids-492c3510511...

I believe we're in for a redesign of OS architectures, now that the 50's assumption of a single machine with a single OS kernel running all programs (or the 60's equivalent of several few mainframes communicating through a slow network) is no longer the main paradigm.

The article reminds me of what Erlang is meant to do. With pure message passing wouldn't the need for some foo() calling some bar() be down to shooting off a message?

https://www.youtube.com/watch?v=o4-YnLpLgtk&t=397s https://www.youtube.com/watch?v=OBfB2MJw3qg

Great write-up and I agree with many points. I don't think the 'Lisp language' is the key point here though, but the other OS properties like a single level store and composition methods richer than byte arrays.

Would be nice if the author referenced historical systems that overlap these ideas.

One is the Symbolics Genera system with 'data level integration' so apps shared rich data structures instead of uninterpreted bits. https://www.ifis.uni-luebeck.de/~moeller/symbolics-info/gene...

Another is Appolo Domain/OS which had a single level store: http://bitsavers.org/pdf/apollo/014962-A00_Domain_OS_Design_...

Yes, absolutely astounding that this article doesn't mention Genera.

If we're going to proceed to list every single address space operating system we've ever heard of, i'll mention SPIN, written in Modula 3 (although i don't know if it did anything clever about sharing data between applications):

http://www-spin.cs.washington.edu/external/overview.html

Whilst difference is good, I am not sure many of the “attack” on the process etc is right. Whilst you can bring anything on old hardware (car and cdr come to mind and those strange intel register format), some gem can be argued instead of rubbish. Process is an isolation, small address is easier to debug if you do not need 64bit address etc. And driver issue ... lisp driver.

Really not sure the whole game.

Makes me think of loper os: http://www.loper-os.org/?p=4

The criticisms may or may not be correct, but I fail to see why a Lisp (and not another language) would be the answer to all of the issues here.

The points for Common Lisp are:

1. It is ANSI standardized and mature.

2. Common Lisp programs are safe. Some implementations violate safety for performance reasons, but this one certainly won't.

3. Common Lisp has the best error handling mechanism that I know of (Conditions, handlers and restarts).

4. The language is programmable, both at the syntax level (Macros) and at the object level (CLOS with the metaobject protocol). So new ideas from other programming languages can often just be ported.

5. The language has been designed for interactivity and safe incremental redefinition.

6. Lisp has already been used very successfully for operating system development (https://en.wikipedia.org/wiki/Lisp_machine).

This makes it pretty much the ideal choice for such a project.

This document talks a lot about deficiencies of the unix way, yet fails to give any concrete examples of advantage of the proposed new way where everything runs in the same address space, uses object storage, and is allowing users to run (!) "only code that has been produced from the high-level notation of the language".

I agree that there are technical problems with interoperability of programs and partially this may be due to the unix way. But the article does not analyze this and it is not clear that such a radical change is a good way to solve those problems. Even if theoretically yes, there are practical things to consider, such as the insane amount of work needed to switch. Perhaps the author just likes Lisp and has a solution in need of a problem.

Does anyone remember the TI Explorer? (https://en.wikipedia.org/wiki/TI_Explorer). They were sold in the 1980s, I first saw one in person in 1993. It was astonishing back then that it had an optical mouse and an optical cable connecting the keyboard to the rest of the unit. My understanding is that it was a full stack LISP machine, down to CAR/CDR support in hardware.

How does failure-recovery (software bugs, hardware, or otherwise) or resilience-engineering more generally work on a system that demurs isolation and redundancy (honest question)?

They spelt emacs wrong

They didn't name anything, they just provided a description.

I assumed his comment was a reference to the old "joke":

> "Emacs is a great OS, but it lacks a decent text editor."

The comment presumably made because emacs is extended with LISP.

Emacs is written in Emacs LISP, which is its own language. The really performance-critical stuff is written in C, as is the interpreter, but that’s it.

That's ostensibly what I said wasn't it? Sorry but I can't tell if you were agreeing or disagreeing with me.

I interpreted your comment to mean something like “Emacs is configurable with third-party LISP code”, in much the same way as Vim is configurable with third-party Vimscript.

But Vim is written in C, Vimscript being just a bolt on, whereas Emacs LISP is its own language entirely, a distinct variant of LISP, and much of Emacs implemented in it.

LISP is a family of languages (much like BASIC). So Emacs Lisp is still a LISP.

I don't even know why you're bring Vim into the discussion when it has nothing to do with the topic being discussed. It doesn't invalidate the statement I made; I just makes your reply look a little snobby because you seem concerned that someone might compare Emacs and Emacs Lisp to Vim and Vim Script (I clearly wasn't given the complete lack of reference to Vim in my post).

To be honest I thought we had gotten past those tedious flamewars in the 90s.

This joke 2.0 would be about ChromeOS and a decent web browser.

There’s lots of great stuff about emacs but it doesn’t really fit the description in the OP.

I literally stalled on the first sentence where he asserts without any rationale that an OS written in Lisp "would be a good thing in itself".

WHY

WHY WOULD ANYONE THINK THIS MATTERS

I would, and it's based on a pretty simple philosophical/moral/spiritual belief:

Anything that increases and improves our knowledge of the universe - including knowledge of how to best use existing tools, and which tools to use for which jobs - is a good thing in and of itself (barring exceptions related to negative externalities which don't apply here).

I don't know if that's the author's rationale, but it's the one I would use.

Conversely, why would anyone think that because a thing doesn't matter to them, it shouldn't matter to anyone else either?

I think that if you were to poll the users of an OS about what matters to them, the language it was implemented in would be at the absolute bottom of the list.

It may matter to the author of the article. That's fine. To the users? I'm pretty sure that hardtoreason correctly reflects their attitude.

And if you're writing an OS, but don't care about the users? Well, write it however matters to you, because you aren't likely to have users who care, because you aren't likely to have users at all.

> And if you're writing an OS, but don't care about the users? Well, write it however matters to you, because you aren't likely to have users who care, because you aren't likely to have users at all.

Counterpoint: Windows 10.

Touche. (Though even with Windows 10, my larger point still stands - nobody cared what language it was written in.)

Reply on news.ycombinator.com