The 3L Project In Depth

Technical Details

3L is really just a single Lisp program. There is only one address space and one runtime that everything else runs inside of. We can do this securely because the language runtime provides and enforces "first class environments". An environment is an object containing references to resources available in the system. It's like running a program but the program doesn't choose what libraries or hardware resources it has access to. Instead programs are run inside their own environment that provides bindings to things it is allowed to use. If you don't want a program to access the network then run it inside of an environment that has no network bindings. If the program calls a function that interacts with the network that function won't be defined in its environment and an exception will be triggered. This mechanism provides for a very robust, fine-grained, comprehensive, and simple security system.

At its core 3L is a Lisp runtime. Beyond the standard lisp features the runtime also provides facilities for hardware and memory access, debugging, introspection, first class environments, and first class source code and documentation manipulation.

In Summary, why rewrite the OS?

The structure and interface of OSes in use today are based mostly on the state of computer science 45 years ago and much more research has been done since then that improves dramatically upon our knowledge at that time. Hardware is also not as limiting of a factor now and is likely to continue improving. The concepts behind this OS is the result of decades of research and development. While implementations of OSes has improved considerably since the first Unix systems were created the fundamental design has changed little. This is not limited to Unix based systems either. All major OSes are built with the same underlying model of an OS kernel being separate from a language runtime and user programs. And they generally use the same tacked-on security measures like user privileges and access-control-lists.

The OSes of today aren't going away for a long, long time but that doesn't mean they are the end-all-and-be-all of the OS. Instead we need to look to the future and build a system based on our new knowledge and understanding.

Why not just improve the OSes we have now? That would be ideal but they are so fundamentally far away from where we need to go that it would have the same net result as starting from scratch, except take longer. That being said, not all is lost. It is still possible to port programs over but how easy that is will depend greatly on how high level they are and how well the language they are written in can deal with the module system and security model. Emulation layers can also be created but should be avoided since they will require an almost unrestricted access to all hardware and system libraries which would completely ruin the improved security infrastructure.

Languages besides lisp

Even though the core of 3L is written in lisp that does not mean other languages won't be available. Transpilers, byte code compilers, and interpreters will be provided for many languages, including popular ones like Javascript, Python, and Ruby. In fact, every language can be support in a safe way and with all or most of the debugging features and introspection available to lisp programs. Even languages like C can be supported via memory access emulation (although those programs will necessarily run slowly).

Dropping the filesystem

The filesystem is not the best way to store every type of data. 3L instead just provides a simple key-value storage mapping. A key is a label that maps to a set of blocks on a storage device. Traditional filesystems can be built on top of this mechanism as simple libraries. Security is provided via libraries and the first class environments. If you want to only allow a program access to a specific set of blocks you can create a function that will only read and write to those blocks and pass that function in to the program you want to utilize it. (A more flexible abstraction is provided in practice.) This means you have 100% control of the storage available. Dropping OS level filesystem control also makes it much easier for things like databases to optimize their read and write operations.

PreScheme

Lisp itself needs runtime support, like a garbage collector, to manage things like anonymous functions. This means that before the full Lisp can be used something else has to set it up. 3L uses "PreScheme" which is a restricted subset of Scheme that provides low-level memory access. Aside from a few things that must be written in assembly the entire Lisp runtime and OS is written in at least a subset of Scheme. The advantage of this, aside from being easier to develop and more concise, is that a full Scheme system can be used for development, including a debugger and profiler. Of course some things, like drivers, must be emulated but much of the code, like standalone functions and libraries, can still be developed on a full Scheme runtime. This greatly speeds up development.

Performance

The safety and programmer facilities come a cost but mechanisms will be provided to over come much of these limitations. You can choose to block compile parts or all of the OS and programs and functions can be integrated and inlined along with many other optimizations enabled by block compilation. One major performance issue that current OSes deal with is interrupt handling which requires a lot of memory manipulation and processing power; 3L does not have to deal with that issue. With those performance enhancements 3L could likely achieve 80-90% the speed of current OSes. (Also, see note on 'unsafe operations' below for another performance improvement possibility.)

Unsafe Operations

The mainline version of 3L does not provide any support for running native code as a part of increasing security and development features.

However, 3L does not architecturally prevent an extension that would allow programs to directly access the memory or run "native" code. It could even be extended to allow programs to run in their own address space. This extension won't be in the default installation but could be provided as a separate version of 3L. This is an option if you want the maximum performance.