[movitz-devel] Lisp OS

[mikel evins]

The reason I downloaded Movitz and joined the list is that I'm 
interested in OS development in Lisp. That interest has its roots in 
two experiences: first, working with a Symbolics machine and Genera in 
the late 80s; second, being a programmer working on an operating system 
at Apple in the early nineties. The operating system was for a new 
platform that eventually became the Newton, and everything except the 
microkernel and the QuickDraw library were written in a new programming 
language called Ralph; Ralph was later renamed Dylan, and later than 
that its syntax was changed to something more familiar to C 
programmers, but Ralph was originally just Scheme plus CLOS.

Working with Genera was a lot of fun, and quite different from any 
other OS I've used. Working on the Dylan version of Newton's OS was 
even more fun.

One of the things that the two had in common was that they were both 
designed to be Lispy; that is, the abstractions and interfaces provided 
were designed to be natural to the Lisp programmer.

There's a lot of code and a lot of documentation available about how to 
write operating systems, but nearly all of it assumes that system 
programmers will use C, and they are organized around that assumption. 
If you assume that the system-programming language is Lisp and you 
think about system programming in Lisp, things turn out a little 
differently.

To take one example, I have experience working with Apple's QuickDraw 
libraries on five different processors (6502, 68K, PowerPC, x86, and 
ARM) and four operating systems (Apple IIGS, MacOS, Windows, and 
NewtonOS). The versions of QuickDraw whose API was presented in C were 
more similar to each other than different, despite different processors 
and operating systems, but when we did a windowing system in Ralph for 
the Newton, based on the lowest-level QuickDraw code, that version of 
QuickDraw was different from the others. QuickDraw had seven very 
low-level routines; all the familiar C API was implemented in terms of 
those low-level routines. We kept the low-level routines, but discarded 
the higher-level API; the guy working on QuickDraw implemented a new 
high-level API that was a better match for Lisp.

This Lisp-based QuickDraw API was more different from any of the 
C-based APIs than any of them were from one another.

As an example of the differences, traditional QuickDraw code copies 
relatively large data structures quite a bit, because various different 
routines need access to them in various different ways, and it becomes 
nearly impossible to tell in a nontrivial application which routine 
should deallocate them. In our incarnation of QuickDraw almost all that 
copying was eliminated because garbage-collection freed us from 
worrying about dangling pointers and memory leaks. The various drawing 
data structures lived as long as they were needed and were 
automatically deallocated when they weren't needed anymore. This 
difference in style amounted to an automatic optimization of memory and 
CPU use that gave us a fast, memory-efficient, great-looking graphics 
system that was easy and natural to use.

This story is a prologue to a discussion I'd like to have about OS 
design in Lisp (and on Movitz): what would a modern OS look like if 
designed around the natural abstractions and expressions of Lisp?

Plan 9 is said to be organized around 3 fundamental concepts:

1 - everything in the system looks like a file
2 - files respond to the same unifying protocol regardless over whether 
they are local or remote
3 - the namespace of the filesystem is subjective; that is, each 
process names all the files in the universe using a single namespace, 
but that namespace is defined relative to the process itself, so two 
different processes may have very different views of the same set of 
files.

These fundamentals give rise to a system with powerful and elegant 
abstractions. What are the corresponding small number of powerful 
fundamentals of a modern Lisp OS? What are the irreducible atoms of 
which it is constructed? What are the abstractions that make for the 
most natural expression of OS ideas in Lisp? What would the ultimately 
Lispy OS look like?