Packages

Introduction

On the one hand, packages solve an important problem: reducing name conflicts between independent code modules. On the other hand, using them incorrectly can cause maddening file loading problems and, paradoxically, increased name conflicts. Most modern languages have something similar, e.g., namespaces in C++ and packages in Java, and all of them confuse novices.

The Name Conflict Problem

The name conflict problem comes up in large system development, no matter what programming language is being used. Large systems are broken into modules, that are developed by independent teams of programmers, and then loaded together to form the final system. This last step is usually called system integration.

Unfortunately, sometimes modules that worked perfectly when tested individually fail to work when integrated. The cause is usually one or more name conflicts, that is, the same name is used for a function or global variable in two different modules in two different ways. The hardest bugs to find are when the functions were intended to be the same but differ slightly, for example, the function in one module is an older version of the one in the other module.

Avoiding this problem involves two steps:

• When the system is broken down into separate modules, the functions and global variables each one "exports" to the whole system are carefully defined, and any potential name conflicts are resolved up front.
• When a module is implemented, any functions and variables defined that are not exported are hidden in some manner.

In Common Lisp, hiding is done with packages.

What are Packages?

Lisp, like other programming languages, distinguishes symbols, like car, from strings, like "car". Symbols are used to name variables and functions. When your code is read, the Lisp reader converts characters into symbols, numbers, strings, and so on, according to the punctuation rules of Lisp. In order for your code to work, the reader has to return the same symbol, e.g., car, every time it reads the same sequence of characters, e.g., c, a and r. To do this, the reader keeps a table of all the symbols it has seen. If a character sequence has not been seen before, a new symbol for it is created, stored in the table, and returned. If it has been seen before, the previously created symbol is returned. This table is called a package.

The current package is determined by the value of the special variable *package*. You can change the current package by either setting this variable, or by executing (in-package package-name).

When you create a library of code, you make sure that the names you use don't conflict with any other code library by defining the library in a new package. You do this with defpackage and in-package, as described below. Then, anyone can use your library by simply loading the code files and typing (use-package package-name). use-package makes a link between the current package and the package named. It does not copy anything.

The reader looks for symbols in the current package, as determined by *package*, then in the packages linked to that package by use-package. If it finds a matching symbol anywhere, it uses that one. Otherwise, it creates a new symbol in the current package.

Symbols in packages are marked as external or internal. When you call use-package, you only get external symbols.

Lisp comes with two packages pre-defined:

• common-lisp contains the symbols for all the variables and functions defined. Any code that uses just the symbols in this package should run in any Common Lisp implementation. Neither you nor any vendor should add any external symbols to this package.
• common-lisp-user is the most common default package. It uses the common-lisp package plus other packages vendors provide with editing and debugging tools. It's also called cl-user for short.

Using Packages

There are two steps to "packaging" a module:

• Creating the new package, using defpackage.
• Starting a file with module code with an in-package form.

defpackage

You can create a package with the following template:

(defpackage #:package-name
  (:use #:common-lisp ...)
  (:export #:symbol1 #:symbol2 ...))

Here's what the above does:

• It defines a package called package-name.
• It lists the packages that this package needs, i.e., packages that define functions used by code in this package. Almost always, you need the common-lisp package.
• It lists the symbols that are external in this package.
• It uses the special reader syntax #:symbol to refer to all symbols.

The last point is easy to do but somewhat obscure to understand. Here's the problem: how do you refer to symbols in a package that don't exist yet? Suppose you wrote

(defpackage foo (:use common-lisp) (:export fun1 fun2))

The Lisp reader will read this form, and create the symbols foo, fun1 and fun2 in the current package. Then the Lisp evaluator would execute this form, create the package foo with the external symbols fun1 and fun2. But look what happened: fun1 and fun2 have been created in the current package. If you try to use the foo package, you'll get a name conflict because fun1 and fun2 exist in both the current package and in foo. But of course you didn't want to put fun1 and fun2 into the current package.

One way to avoid this problem is this:

(defpackage "FOO" (:use "COMMON-LISP") (:export "FUN1" "FUN2"))

This creates no symbols at read-time. It only uses name strings for symbols, not symbols. You have to use uppercase because standard Lisp stores all symbols with uppercase names. The standard Lisp reader is case insensitive. It converts all symbol characters to uppercase internally.

Unfortunately, the above won't work in what Franz calls the "modern" version of Allegro. This version uses a case sensitive reader to support linking with Java, which uses case-sensitive names. The version stores standard Lisp names in lowercase. To make the above code work, you'd have to write "fun1" and "fun2" above, but then you'd have code that wouldn't work in standard Common Lisp.

A common alternative is to write this:

(defpackage :foo (:use :common-lisp) (:export :fun1 :fun2))

This uses keyword symbols. defpackage uses the internal symbol name strings so in standard Common Lisp, it will get "FUN1" and "FUN2" and in "modern" Allegro it will get "fun1" and "fun2". This works but adds every symbol in two places: the package being defined and the keyword package.

The #: approach solves this problem. #: tells the Lisp reader to create a symbol but not put it in any package. As soon as the symbol's not used anywhere, it will be garbage-collected.

Package local nicknames

Modules and packages are an area where modern Lisp practice has evolved beyond the standard, as the use of libraries has increased, in keeping with other languages such as Python and JavaScript. Now almost everyone uses ASDF and Quicklisp, to load libraries as needed, even though ASDF is not in the standard.

Similarly, many argue that the :use keyword should not be used in defpackage except to import the Common Lisp package and local internal packages. :use with an external library means that your code is open to future name conflicts if new names are added and exported by that library.

Instead of :use, the recommendation is to explicitly include the package name on all external library symbols, e.g., don't write:

(defpackage #:my-package
  (:use #:common-lisp #:alexandria))

(defun my-function (n) (map-iota (lambda (x) (* x x) n)))

Instead write

(defpackage #:my-package
  (:use #:common-lisp #:alexandria))

(defun my-function (n) (alexandria:map-iota (lambda (x) (* x x) n)))

This code is safe from new names being added to the Alexandria library, but the code can become harder to read when libraries have long names, such as alexandria. Long names are good to avoid conflicting package names.

For this reason, the keyword :local-nicknames has been added to defpackage. This is not in the standard, but all major Common Lisps now support it. It lets you specify a nickname for a package that is only defined in the package you are creating. This lets you specify very short nicknames for the packages you are using.

(defpackage #:my-package
  (:use #:common-lisp)
  (:local-nicknames (:a #:alexandria)))

(defun my-function (n) (a:map-iota (lambda (x) (* x x) n)))

This also makes it clearer in the code when external symbols are being used.

User packages versus code packages

common-lisp is an example of a code package, i.e., a package intended to hold a library of useful functions. common-lisp-user is an example of a user package. Allegro for Windows defines many code packages, and a user package called cg-user. For this course, there are many code packages, such as frames and mops, and a user package cs325-user.

A user package is intended for interactive use in a Lisp listener window. You create a user package, link it to code packages, and make it the current package. A user package lets you control which code packages are present, hides the internal symbols in the code packages, and protects the code packages from accidental changes.

User packages are treated differently than code packages in two ways:

• You don't export anything from a user package. You use in-package to switch into the package to do work.
• You don't use-package a user package. That's a recipe for accidental and unnecessary name conflicts.

Where to create packages

The Hyperspec recommends putting all package definitions in a single file. This works well with ASDF systems. This file is typically called package.lisp. By loading that package file before anything else, you don't run into any timing issues.

For one-off utility files that don't merit an ASDF system, I put the defpackage at the front of the file. A programmer can simply load that file and not have to worry about package definition. Note that if the package definition uses packages other than common-lisp, you need to make sure those packages have already been loaded. If that's the case, the first approach is better.

in-package

The very first form in a file (or second, if there is a defpackage) should be:

(in-package #:package-name)

This tells Lisp, the Lisp editor, if any, and all Lisp programmers, what package the code in this file is in. Some Lisp implementations will warn you if a file doesn't start with in-package. Others will assume you meant to say (in-package #:common-lisp-user).

There are two simple good programming rules to follow:

• Always define a package because calling in-package.
• Use only one in-package per file.

Classic Mistakes with Packages

Bad Timing

Has something like this ever happened to you?

> (load "tables")
> (deftable find-rule)
Error: Undefined function: find-rule...
> (use-package #:tables)
Error: Name conflict: deftable already exists in common-lisp-user

Here's what happened:

• Lisp loaded tables, created the tables package, but didn't automatically connect that package to the current package. (Nor should it, since this might not be what you want.)
• Lisp read (deftable find-rule). Since there was no deftable in the current package, the reader made one and added it to the current package. But of course this new symbol had no function attached, so an error occurred.
• Calling use-package caused Lisp to try and connect the current package to tables. It properly noticed that you already have a deftable, namely the one it just created in the previous expression.

First, you need to get into the habit of calling use-package immediately after load or require.

Second, you can recover by using unintern to remove the offending symbols and calling use-package again. Many Lisp environments offer an automatic unintern as an error response option.

Using in-package instead of use-package

Do you do this?

> (load "mops")
> (in-package #:mops)
> (load "my-code")
> ...test stuff...

If you have, then you've missed the whole point of packages.

Packages are supposed to avoid name conflicts by hiding internal function and variable names. But if you do an in-package, you put your code into the package in question, thereby guaranteeing that if you use any of the same names, you will clobber the package's code.

in-package should only appear at the front of files that define or extend an existing package.

Use use-package to make the exported symbols of packages you need available to your code. Use in-package at the Listener window only to switch into a user package.

Linking to User Packages

Have you done something like this?

(defpackage #:rules
  (:use #:common-lisp-user ...))

If you have, then the odds are good that your code will not be portable. In particular, when you load this code into another Common Lisp system, you will get a name conflict, or unexpected "undefined function" errors.

common-lisp-user is a working environment. It gathers together a number of packages, provides a home for temporary test variables, and so on. What's in common-lisp-user varies from implementation to implementation, and even from one release to the next. The only guarantee is that common-lisp-user will include the common-lisp package.

If your package uses common-lisp-user, two problems can arise:

• Your packages defines a function, say show-window with no conflict in your Lisp system. In another system, it turns out that common-lisp-user uses a graphics package that also defines show-window. Your code will accidentally redefine the built-in function.
• Your code uses a function, say show-window, which you mistakenly believe is standard Common Lisp. In fact, it's there because your common-lisp-user uses a graphics package. When your code is loaded into another system, it breaks with undefined function show-window.

Any package you create should use common-lisp (lisp in Common Lisp 1), plus any other non-user packages it needs. That way, you know exactly which functions you're getting and which ones your code depends on.