Modularizing Code in Common Lisp

Not that many years ago, using other people's Common Lisp libraries was a pretty tedious affair. You had to download their code files, see what other code files were required, hope there weren't any incompatibilities, etc. Fortunately, these days we have Quicklisp to automate most of the process. Quicklisp in turn uses ASDF, a non-standard but popular tool for defining systems, i.e., collections of files and how they depend on each other and other systems.

Quicklisp

Quicklisp is a library manager, similar in some ways to Ruby's gem system. Once installed, Quicklisp makes it easy to

To do this, Quicklisp has a server full of Common Lisp code, but it also needs to know what files and libraries each library needs. To do that, it uses ASDF.

ASDF

Common Lisp has no standard for libraries. There have been many implementations of ideas for defining libraries. Mark Kantrowitz's DEFSYSTEM was one of the more popular early ones. It used the term "system" for "library." It was succeeded by ASDF (Another System Definition Facility). The current version is ASDF 3. Many Lisps have ASDF installed, though it might be version 2.

ASDF definitions are (very) roughly like Unix makefiles. They describe what libraries and files needs to be compiled and loaded to make a system, and what the file dependencies are. If A depends on B and B changes, A should be recompiled as well.

ASDF definitions can get quite complicated. For our purposes, we're going to use the simplest form. Here's the ASDF definition for the CS325 code library:

(asdf:defsystem #:cs325
        :serial t
        :depends-on (#:lisp-unit)
        :components ((:file "package")
                    (:file "tables")
                    (:file "extend-match")
                    (:file "lisp-critic")
                    (:file "ddr")
                    (:file "exercise-tests")
                    (:file "lisp-rules")
                    (:file "ddr-tests")
                    (:file "ddr-exs-tests")
                    ))

This says that the system is named cs325. That means you can load this with (ql:quickload "cs325").

The cs325 system consists of 12 files. The :serial t says that there's a linear (serial) dependency: later files depend on earlier files. So, if lisp-critic.lisp changes, it and every file after it in the list will be recompiled and reloaded the next time (ql:quickload "cs325") is called.

This isn't really necessary. Most of these files only depend on dependencies.lisp and package.lisp. A few depend on tables.lisp. lisp-rules.lisp depends on lisp-critic.lisp, etc. As a result, if, say, tables.lisp is modified, more files will be compiled and reloaded than necessary. It's not hard to change the ASDF to do the right thing, but we won't bother since there's not that many files and they all compile quite quickly.

The cs325 system doesn't depend on any libraries, but the JSON Demo system does.

(asdf:defsystem #:json-demo
      :serial t
      :depends-on ("simple-server" "cl-json")
      :components ((:file "json-demo")))

The :depends-on clause lists other systems that are needed, before the files are loaded. In this case, there's one local system simple-server and a Quicklisp-managed JSON library cl-json. CL-JSON in turn depends on many other libraries, that Quicklisp will automatically download, compile and load.

Local Systems

You add a local project to Quicklisp by putting a directory of code inside ~/quicklisp/local-projects/ containing a file with the extension asd that has the ASDF definition of your system. (ql:register-local-projects) will recursively scan all the directories in local-projects for .asd files and put them in a list in ~/quicklisp/local-projects/system-index.txt. ql:quickload checks this file first before looking for modules online.

SBCL Windows note: SBCL requires that text files has Unix line endings, even on Windows. Quicklisp will break reading the local projects file if the line endings are Windows-style. This can happen if you edit the file by hand, or call (ql:register-local-projects) in a different Lisp on your machine.

Readings

Phil Eaton's Starting a minimal Common Lisp project is a good introduction to modern Common Lisp library packaging with ASDF and Quicklisp.