Macro functions are the means by which users can define forms that behave very much like Common Lisp's special forms. Many Common Lisp special functions, such as let and cond, could be defined as macros, assuming the existence of more primitive forms, such as lambda and if.

What are macros good for?

Far and away, the most common need for macros is to implement definers. Definers usually have names that begin with "def" or "define-" and are used to attach data to names. defun, defstruct, defclass, and so on, are examples of definers already in Common Lisp.

Here are two examples from the CS 325 code library:

(defmop m-elephant (m-animal)
    :color m-grey)
(define-test pick-greater
   (assert-equal 5 (pick-greater 2 5))
   (assert-equal 5 (pick-greater 5 2))
   (assert-equal 10 (pick-greater 10 10))
   (assert-equal 0 (pick-greater -5 0))
   )

Why can't you implement these as functions?[1]

Another common macro type is the with- macro. Built-in examples are with-open-file, with-input-from-string, with-output-to-string, and with-accessors. These macros usually have the form (with-... (variable specifications) body of code), e.g.,

(with-input-from-string (in "1 2 3")
  (list (read in) (read in) (read in)))

Occasionally, though less often than you might think, there are macros for iteration. The list-of exercise is one example. It lets you construct, for example, "the list of all n in a list that are odd" like this:

(list-of (n :in '(1 2 3 4 5)) (oddp n))

For many people, definers are the only kinds of macros they will need to create. Fortunately, they are also the simplest to implement, once you understand how macros work.

How do macros work?

A macro call looks just like a function call, but the Lisp interpreter has the following special rules for evaluating macro calls:

  1. The macro function is applied (using apply) to the argument forms. Note: the arguments are not evaluated. The value returned is called the expansion of the macro.
  2. The expansion is evaluated.

The Lisp compiler has similar rules:

  1. The macro function is applied (using apply) to the argument forms.
  2. The expansion is compiled.

Because the compiler has to be able to expand a macro call at compile time, it makes no sense for a macro definition to depend on the run-time value of a variable.

How are macros defined?

A macro definition looks just like a function definition, except that you use defmacro instead of defun. E.g.,

(defmacro macro-name (parameters) ...)

The rules for evaluating macros imply a very important, but often forgotten, principle for macro definitions:

A macro function should not do things. It should construct code to do things.

For example, suppose we wanted

(test-exp expression)

to print a message of the form

expression => value

e.g.,

(let ((x 1) (y 2)) (test-exp (+ x y)))

would print

(+ x y) => 3

Why can't test-exp be a function?[2]

Here is a possible definition of the macro test-exp:

(defmacro test-exp (exp)
  (format t "~&~S => ~S~%" exp (eval exp)))

What's wrong with this definition? When would it fail?[3]

Here's the proper definition:

(defmacro test-exp (exp)
  `(format t "~&~S => ~S~%" ',exp ,exp))

Notice that all the macro function does is build a piece of code. That's all any macro should do.

The Macro Defining Method

The following method for defining macros is strongly recommended. It's designed to avoid:

Here's the method:

  1. Define a function that does the "real" work. (This does not usually apply to control-flow macros, like looping forms.)
  2. Write down several examples of how the macro would be used.
  3. For each example, write down the equivalent code you'd write, using the function defined in Step 1, if the macro didn't exist.
  4. (a) Write the defmacro header, (b) insert one of the function call examples as the body, and (c) "parameterize" it, using backquote and "comma" and "at-comma" forms to replace specific data items with macro parameters.
  5. Test the definition on the other examples and tweak as needed.

Implementing the macro test-exp

Step 1: Possible, but not really necessary.

Step 2: Write down some examples:

(test-exp (+ x y))

Step 3: Write down the equivalent code without the macro:

(format t "~&~S => ~S~%" '(+ x y) (+ x y))

Step 4(a): Write the defmacro header. :

(defmacro test-exp (form)
  ...)

Step 4(b): Insert one of the examples in the body:

(defmacro test-exp (form)
  `(format t "~&~S => ~S~%" '(+ x y) (+ x y)))

Step 4(c): Parameterize the body with backquote, etc. That is, we add a backquote, and replace specific data items with the macro's parameters. Thus, all occurrences of (+ x y) become ,form. Notice that the quote's remain in the macro body. 

(defmacro test-exp (form)
  `(format t "~&~S => ~S~%" ',form ,form))

Step 5: Test on other examples.

Implementing the macro defmop

Let's use the method to define defmop.

Step 1: define a function to do the real work, e.g.,

(defun add-mop (...) ...)

We have to decide how this function would be called and what it does. Mops have a name, a list of abstractions, and a list of slots, and they're stored under the name. The principle of "Short Attention Span Programming" dictates that separate tasks be implemented in separate functions, so we'll define make-mop to make a mop structure, and store-mop to save that structure under its name in a table.

(defun add-mop (name absts slots)
  (store-mop name
     (make-mop :name name
               :absts absts
               :slots slots)))

Step 2: write down the some defmop calls:

(defmop m-elephant (m-animal)
  :color m-grey)
 
(defmop m-animal (m-thing))

Step 3: write down the equivalent calls to add-mop:

(add-mop 'm-elephant '(m-animal)
         '(:color m-grey))
 
(add-mop 'm-animal '(m-thing) '())

Step 4(a): Write the defmacro header. From our examples, we know that we want defmop to take a name, a list of abstractions, and then zero or more slots, so the header would be: 

(defmacro defmop (name absts &rest slots)
  ...)

Step 4(b): Insert one of the examples in the body:

(defmacro defmop (name absts &rest slots)
  (add-mop 'm-elephant '(m-animal)
           '(:color m-grey)))

Step 4(c): Parameterize the body with backquote, etc. That is, we add a backquote, and replace specific data items with the macro's parameters. Thus, m-elephant becomes ,name, (m-animal) becomes ,,absts, and (:color m-grey) becomes ,slots. Notice that the quotes are not part of the specific data, so they remain in the macro body.

(defmacro defmop (name absts &rest slots)
  `(add-mop ',name ',absts
            ',slots))

Step 5: Test on other examples. E.g., does (defmop m-animal (m-thing)) work with the above definition, or does it need tweaking?

Debugging Macros

Don't rely on what the macro call returns when debugging. Use the following idiom to see what a macro call expands into. If the expansion looks suspicious, it probably is wrong, even if the right answer comes back in some test cases:

(pprint (macroexpand-1 'macro-call))

Defining Complex Macros

There are two ways in which a macro can be too complex for Step 4 in the above method:

Using Expander Functions

Some macros expand into a form that has to be "calculated" from their arguments. The most common situation is a macro that expands into a set of nested subforms.

A very simple example of this is the macro compose. compose is an extension of function. It takes zero or more function specifications (usually function names, but they could be (lambda ...) forms), and "composes" them into one function. That is,

(compose fn1 fn2 fn3 ...)

returns a function fn with the property that

(funcall fn expression)

is the same as

(funcall fn1
  (funcall fn2
   (funcall fn3 ... expression)))

For example,

> (mapcar (compose 1+ car) '((1 a) (2 b)))
(2 3)

What function should (compose) -- i.e., no arguments -- return?[4]

compose is not a hard macro to define, but clearly it doesn't fit a fixed template. Because of the nested nature of the expansion, it can be most naturally defined with recursion.

When an expansion is complex, define one or more expander functions to construct the expansion the macro needs. Being normal functions, they're easy to call recursively, and easy to debug, too.

Here's a simple definition for compose using an expander function:

(defmacro compose (&rest fns)
  (let ((var (gensym)))
   `#'(lambda (,var)
       ,(expand-compose fns var))))
 
(defun expand-compose (fns var)
  (if (null fns) var
    `(,(first fns)
      ,(expand-compose (rest fns) var))))

Using Argument Parsing Functions

The other way a macro can be complex is if it has a complicated calling format, or if a fair amount of processing has to be done to the arguments. For example, loop takes many different clauses, and each clause has its own format, including,

for variable in list
for variable from start [to end] [by increment]
for variable = initial [then subsequent]
when test clause
finally expression

Another example is bind, defined in AI Programming. 

(bind ((place1 value2)
       (place2 value2)
       ...)
  exp1 exp2 ...)

This generalizes let. Each place is assigned the corresponding value, using setf, then the expressions are evaluated, then place's are restored to their original values. bind generates code with unwind-protect to guarantee that the old values are restored, even if there's a non-local exit (becaues of an error or a call to return-from or throw) during the expression evaluation.

Although the calling format is simple (the same as let), and the expansion fits a fixed format, it turns out bind has to calculate a number of special values from the input arguments before it can fill out the expansion form.

When the processing before expansion is complex, define one or more "parser" functions to process the arguments and get the data needed to build the expansion. Name these functions get-... or parse-..., to keep them straight from the expander functions.

For example, here's a cleaner definition of the final version of bind in Section 3.16 of AI Programming.

Macros and Maintainable Code

Macros, like functions, should be used when they make code more maintainable, by making it easier to write code that is:

For example, the form

(defmop m-elephant (m-animal)
  :color m-grey)

is more readable than the equivalent function call with the quotes and extra parentheses. The form

(loop for x in l when (test x) collect x)

will usually compile into more efficient code than the standard alternatives. The form

(with-open-file (instream file)
  (do ((x (read instream nil nil)...))
    ...))

generates more robust code than most programmers would write, using unwind-protect to guarantee that streams are closed, no matter what happens.

Macro Pitfalls

Even though macros are a simple idea, and macro definitions use regular old Lisp list functions, there are still several pitfalls that novices fall into repeatedly when defining macros. These mistakes happen when you fail to keep in mind two key facts about macros:

Here are some common macro mistakes. See if you can see what's wrong with each one.

In-lining with Macros

Some Lisps before Common Lisp did not include functions like first, second and so on. So programmers who liked those names, would sometimes define these macros:

(defmacro first (l) `(car ,l))
(defmacro second (l) `(cadr ,l))
...

Programmers would use macros instead of normal functions for efficiency. When compiled, first, second and so on, would be replaced with car, cadr and soon, so the resulting compiled code would be just as efficient as if car, cadr, and so on, had been used

What's wrong with the above code?

Macros are not functions. Therefore, even though (first lst) looks like a normal function call, you'll get an error if you try to do (mapcar #'first lst) or (find x lst :key #'first).

Functions are wonderfully useful things. If something can be a function, it should be a function. The right way in Common Lisp to get efficient function calls, when and only when there's an efficiency problem, is to use the inline declaration. So, if there was a bottleneck in code (highly unlikely) because first, second, and so on were functions, then do this:

(declaim (inline first second ...)
(defun first (l) (car l))
(defun second (l) (cadr l))
...

Reminder: Common Lisp already defines these functions for you.

Problems with macros that look like but are not functions often show up pretty quickly as "... is not a function" type messages. However, "function-like" macros can lead to bugs that are quite hard to find.

A Bad push Macro

Some Lisps before Common Lisp did not include push. Here's a simple definition for (push item list), renamed to x-push to avoid name conflict: 

(defmacro x-push (x l)
  `(setf ,l (cons ,x ,l)))

This will correctly handle (x-push 'a l), (x-push (car x) l), and even (x-push x (car l)). The last form correctly pushes the value of x onto the front of the list stored in the car of l.

What's wrong with the above code?

Simple though it is, the above definition manages to violate two expectations about normal argument evaluation:

Try running the following code. It takes a list of lists, l, and a list of items, e.g., (a b c), and pushes each item onto the corresponding list. (There are better ways to do this, but this is one of the simplest cases I can think of to show the bug that could occur is more complicated code.)

(let ((l (list nil nil nil))
      (i -1))
  (dolist (x '(a b c))
    (push x (nth (incf i) l)))
  l)

If you run the above, you should get ((A) (B) (C)) as a result.

Now try it with x-push. What happens? Why? How could you fix it?

A Bad pop Macro

Some Lisps before Common Lisp did not include pop. Here's a simple definition for (pop list), renamed to x-pop to avoid name conflict. The code has to save the first element of list so that it can return it after resetting list to its tail.

(defmacro x-pop (l)
  `(let ((save (car ,l)))
     (setf ,l (cdr ,l))
     save))

This code has the same multiple evaluation problem that x-push has, but let's ignore that for now. Here's a test of x-pop that seems to do fine.

(let ((l '(a b c)))
  (print (x-pop l))
  (print (x-pop l))
  (print (x-pop l))
  l)

What's wrong with the above definition?

Trying running the test code above, but change the name of the local variable from l to save. What happens? Why? How can you fix it?

Tell me more

For an excellent in-depth treatment of macros, see Paul Graham's On Lisp (Prentice-Hall). It gives a more sophisticated variant of the macro definition methodology outlined above, many examples, caveats, and so on. Warning: the later chapters introduce uses of macros that may be hazardous to code maintenance!

Footnotes

[1] These can't be functions because one or more arguments are not evaluated. [return]

[2] If test-exp was a function, it would be passed 3, the value of (+ x y), not the form itself. [return]

[3] Consider this use of test-exp: 

(let ((x 1) (y 2)) (test-exp (+ x y)))

The call to (test-exp (+ x y)) would expand into 

(format t "~&~S => ~S~%" exp (eval exp))

At run time, exp will have the value (+ x y). The first problem is that (eval '(+ x y)) evaluates the sum of the special bindings of x and y, not the local ones. If there are no such bindings, there will be an error. If you make x and y global, you're still in trouble. The value of format is always nil. Hence (test-exp (+ x y)) expands into nil. So, if you compile a file with the above call to expression, the file will contain 

(let ((x 1) (y 2)) nil)

Not very useful. [return]

[4] (compose) should return a function that simply returns the argument passed to it, i.e., the identity function. It's left as an exercise for the reader to determine if that's what the code given for compose actually does. [return]

Comments? Send mail to Chris Riesbeck.