Introduction

On the one hand, packages solve an important problem: reducing name conflicts between independent code modules. On the other hand, using them incorrectly can cause maddening file loading problems and, paradoxically, increased name conflicts. Most modern languages have something similar, e.g., namespaces in C++ and packages in Java, and all of them confuse novices.

The Name Conflict Problem

The name conflict problem comes up in large system development, no matter what programming language is being used. Large systems are broken into modules, that are developed by independent teams of programmers, and then loaded together to form the final system. This last step is usually called system integration.

Unfortunately, sometimes modules that worked perfectly when tested individually fail to work when integrated. The cause is usually one or more name conflicts, that is, the same name is used for a function or global variable in two different modules in two different ways. The hardest bugs to find are when the functions were intended to be the same but differ slightly, for example, the function in one module is an older version of the one in the other module.

Avoiding this problem involves two steps:

In Common Lisp, hiding is done with packages.

What are Packages?

Lisp, like other programming languages, distinguishes symbols, like car, from strings, like "car". Symbols are used to name variables and functions. When your code is read, the Lisp reader converts characters into symbols, numbers, strings, and so on, according to the punctuation rules of Lisp. In order for your code to work, the reader has to return the same symbol, e.g., car, every time it reads the same sequence of characters, e.g., c, a and r. To do this, the reader keeps a table of all the symbols it has seen. If a character sequence has not been seen before, a new symbol for it is created, stored in the table, and returned. If it has been seen before, the previously created symbol is returned. This table is called a package.

The current package is determined by the value of the special variable *package*. You can change the current package by either setting this variable, or by executing (in-package package-name).

When you create a library of code, you make sure that the names you use don't conflict with any other code library by defining the library in a new package. You do this with defpackage and in-package, as described below. Then, anyone can use your library by simply loading the code files and typing (use-package package-name). use-package makes a link between the current package and the package named. It does not copy anything.

The reader looks for symbols in the current package, as determined by *package*, then in the packages linked to that package by use-package. If it finds a matching symbol anywhere, it uses that one. Otherwise, it creates a new symbol in the current package.

Symbols in packages are marked as external or internal. When you call use-package, you only get external symbols.

Lisp comes with two packages pre-defined:

Using Packages

There are two steps to "packaging" a module:

defpackage

You can create a package with the following template:

(defpackage #:package-name
  (:use #:common-lisp ...)
  (:export #:symbol1 #:symbol2 ...))

Here's what the above does:

The last point is easy to do but somewhat obscure to understand. Here's the problem: how do you refer to symbols in a package that don't exist yet? Suppose you wrote

(defpackage foo (:use common-lisp) (:export fun1 fun2))

The Lisp reader will read this form, and create the symbols foo, fun1 and fun2 in the current package. Then the Lisp evaluator would execute this form, create the package foo with the external symbols fun1 and fun2. But look what happened: fun1 and fun2 have been created in the current package. If you try to use the foo package, you'll get a name conflict because fun1 and fun2 exist in both the current package and in foo. But of course you didn't want to put fun1 and fun2 into the current package.

One way to avoid this problem is this:

(defpackage "FOO" (:use "COMMON-LISP") (:export "FUN1" "FUN2"))

This creates no symbols at read-time. It only uses name strings for symbols, not symbols. You have to use uppercase because standard Lisp stores all symbols with uppercase names. The standard Lisp reader is case insensitive. It converts all symbol characters to uppercase internally.

Unfortunately, the above won't work in what Franz calls the "modern" version of Allegro. This version uses a case sensitive reader to support linking with Java, which uses case-sensitive names. The version stores standard Lisp names in lowercase. To make the above code work, you'd have to write "fun1" and "fun2" above, but then you'd have code that wouldn't work in standard Common Lisp.

A common alternative is to write this:

(defpackage :foo (:use :common-lisp) (:export :fun1 :fun2))

This uses keyword symbols. defpackage uses the internal symbol name strings so in standard Common Lisp, it will get "FUN1" and "FUN2" and in "modern" Allegro it will get "fun1" and "fun2". This works but adds every symbol in two places: the package being defined and the keyword package.

The #: approach solves this problem. #: tells the Lisp reader to create a symbol but not put it in any package. As soon as the symbol's not used anywhere, it will be garbage-collected.

User packages versus code packages

common-lisp is an example of a code package, i.e., a package intended to hold a library of useful functions. common-lisp-user is an example of a user package. Allegro for Windows defines many code packages, and a user package called cg-user. For this course, there are many code packages, such as frames and mops, and a user package cs325-user.

A user package is intended for interactive use in a Lisp listener window. You create a user package, link it to code packages, and make it the current package. A user package lets you control which code packages are present, hides the internal symbols in the code packages, and protects the code packages from accidental changes.

User packages are treated differently than code packages in two ways:

Where to create packages

The Hyperspec recommends putting all package definitions in a single file. This works well with ASDF systems. This file is typically called package.lisp. By loading that package file before anything else, you don't run into any timing issues.

For one-off utility files that don't merit an ASDF system, I put the defpackage at the front of the file. A programmer can simply load that file and not have to worry about package definition. Note that if the package definition uses packages other than common-lisp, you need to make sure those packages have already been loaded. If that's the case, the first approach is better.

in-package

The very first form in a file (or second, if there is a defpackage) should be:

(in-package #:package-name)

This tells Lisp, the Lisp editor, if any, and all Lisp programmers, what package the code in this file is in. Some Lisp implementations will warn you if a file doesn't start with in-package. Others will assume you meant to say (in-package #:common-lisp-user).

There are two simple good programming rules to follow:

Classic Mistakes with Packages

Bad Timing

Has something like this ever happened to you?

> (load "tables")
> (deftable find-rule)
Error: Undefined function: find-rule...
> (use-package #:tables)
Error: Name conflict: deftable already exists in common-lisp-user

Here's what happened:

First, you need to get into the habit of calling use-package immediately after load or require.

Second, you can recover by using unintern to remove the offending symbols and calling use-package again. Many Lisp environments offer an automatic unintern as an error response option.

Using in-package instead of use-package

Do you do this?

> (load "mops")
> (in-package #:mops)
> (load "my-code")
> ...test stuff...

If you have, then you've missed the whole point of packages.

Packages are supposed to avoid name conflicts by hiding internal function and variable names. But if you do an in-package, you put your code into the package in question, thereby guaranteeing that if you use any of the same names, you will clobber the package's code.

in-package should only appear at the front of files that define or extend an existing package.

Use use-package to make the exported symbols of packages you need available to your code. Use in-package at the Listener window only to switch into a user package.

Linking to User Packages

Have you done something like this?

(defpackage #:rules
   (:use #:common-lisp-user ...))

If you have, then the odds are good that your code will not be portable. In particular, when you load this code into another Common Lisp system, you will get a name conflict, or unexpected "undefined function" errors.

common-lisp-user is a working environment. It gathers together a number of packages, provides a home for temporary test variables, and so on. What's in common-lisp-user varies from implementation to implementation, and even from one release to the next. The only guarantee is that common-lisp-user will include the common-lisp package.

If your package uses common-lisp-user, two problems can arise:

Any package you create should use common-lisp (lisp in Common Lisp 1), plus any other non-user packages it needs. That way, you know exactly which functions you're getting and which ones your code depends on.

Faculty: Chris Riesbeck
Time: Monday, Wednesday, Friday: 1pm - 2pm
Location:Annenberg G15

Contents

Important Links