lisp-unit is a Common Lisp library that supports unit testing. There is a long history of testing packages in Lisp, usually called "regression" testers. More recent packages in Lisp and other languages have been inspired by JUnit for Java. For more information on both unit testing and JUnit, visit www.junit.org.
This page has two parts:
My main goal for lisp-unit was to make it simple to use, particularly for beginning Lisp programmers. The advantages of lisp-unit are:
Any test failures will be printed, along with a summary of how many tests were run, how many passed, and how many failed.
You define a test with
(define-test name exp1 exp2 ...)
This defines a test called name. The expressions can be anything, but typically most will be assertion forms.
Tests can be defined before the code they test, even if they're testing macros. This is to support test-first programming.
After defining your tests and the code they test, run the tests with
This runs every test defined in the current package. To run just certain specific tests, use
(run-tests name1 name2 ...)
(run-tests greater summit).
The following example
pick-greaterreturns the larger of two arguments
First, we define some tests.
> (in-package :cs325-user) #<PACKAGE CS325-USER> > (define-test pick-greater (assert-equal 5 (pick-greater 2 5)) (assert-equal 5 (pick-greater 5 2)) (assert-equal 10 (pick-greater 10 10)) (assert-equal 0 (pick-greater -5 0)) ) PICK-GREATER
Following good test-first programming practice, we run these tests before writing any code.
> (run-tests pick-greater) PICK-GREATER: Undefined function PICK-GREATER called with arguments (2 5).
This shows that we need to do some work. So we define our
broken version of
> (defun pick-greater (x y) x) ;; deliberately wrong PICK-GREATER
Now we run the tests again:
> (run-tests pick-greater) PICK-GREATER: (PICK-GREATER 2 5) failed: Expected 5 but saw 2 PICK-GREATER: (PICK-GREATER -5 0) failed: Expected 0 but saw -5 PICK-GREATER: 2 assertions passed, 2 failed.
This shows two failures. In both cases, the equality test
returned NIL. In the first case it was because
(pick-greater 2 5) returned 2 when 5 was expected, and
in the second case, it was because
(pick-greater -5 0) returned -5 when 0 was expected.
The most commonly used assertion form is
(assert-equal value form)
This tallies a failure if form returns a value
equal to value. Both
value and test are
evaluated in the local lexical environment. This means
that you can use local variables in tests. In particular,
you can write loops that run many tests at once:
> (define-test my-sqrt (dotimes (i 5) (assert-equal i (my-sqrt (* i i))))) MY-SQRT > (defun my-sqrt (n) (/ n 2)) ;; wrong!! > (run-tests my-sqrt) MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8 MY-SQRT: 2 assertions passed, 3 failed.
However, the above output doesn't tell us for which values of
the code failed. Fortunately, you can fix this by
adding expressions at the end of the
These expression and their values will be printed on failure.
> (define-test my-sqrt (dotimes (i 5) (assert-equal i (my-sqrt (* i i)) i))) ;; added i at the end MY-SQRT > (run-tests my-sqrt) MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2 I => 1 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2 I => 3 MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8 I => 4 MY-SQRT: 2 assertions passed, 3 failed.
The next most useful assertion form is
This tallies a failure if test returns false. Again, if you need to print out extra information, just add expressions after test.
There are also assertion forms to test what code prints, what errors code returns, or what a macro expands into. A complete list of assertion forms is in the reference section.
Do not confuse
with Common Lisp's
is used in code to guarantee that some condition is true. If it isn't,
the code halts.
assert has options you can use
to let a user fix what's wrong and resume execution. A similar collision
of names exists in JUnit and Java.
Tests are grouped internally by the current package, so that a set of tests can be defined for one package of code without interfering with tests for other packages.
If your code is being defined in
which is common when learning Common Lisp, but not for
production-level code, then you should define your tests
cl-user as well.
If your code is being defined in its own package, you should define your tests either in that same package, or in another package for test code. The latter approach has the advantage of making sure that your tests have access to only the exported symbols of your code package.
For example, if you were defining a date package,
date.lisp file would look like this:
(defpackage :date (:use :common-lisp) (:export #:date->string #:string->date)) (in-package :date) (defun date->string (date) ...) (defun string->date (string) ...)
date-tests.lisp file would look like this:
(defpackage :date-tests (:use :common-lisp :lisp-unit :date)) (in-package :date-tests) (define-test date->string (assert-true (string= ... (date->string ...))) ...) ...
You could then run all your date tests in the test package:
(in-package :date-tests) (run-tests)
Alternately, you could run all your date tests from any package with:
Here is a list of the functions and macros exported by lisp-unit.
*package*in effect when
define-testis executed. The expresssions are assembled into runnable code whenever needed by
run-tests. Hence you can define or redefine macros without reloading tests using those macros.
nilis given, it removes all tests for all packages.
*package*in effect when the macro is expanded. If no names are given, all tests for that package are run.
All of the assertion forms are macros. They tally a failure if the associated predication returns false. Assertions can be made about return values, printed output, macro expansions, and even expected errors. Assertion form arguments are evaluated in the local lexical environment.
All assertion forms allow you to include additional expressions at the end of the form. These expressions and their values will be printed only when the test fails.
Return values are unspecified for all assertion forms.
These macros tally a failure if value is not equal to the result returned by form, using the specified equality predicate.
is used for most tests. But any binary predicate can be used, with
(assert-equality #'unordered-equal '(a b c) (unique-atoms '((b c) a ((b a) c))))
assert-truetallies a failure if test returns false.
assert-falsetallies a failure if test returns true.
(macroexpand-1 form)does not produce a value equal to expansion.
errorto refer to any kind of error. See condition types in the Common Lisp Hyperspec for other possible names. For example,
(assert-error 'arithmetic-error (foo 0))would assert that
foois supposed to signal an arithmetic error when passed zero.
(when (> (length queue) 100) (fail "Queue exceeded expected size by " (- (length queue) 100)))
Several predicate functions are exported that are often useful in
writing tests with
can be used to specify an equality predicate. The default is
can be used to specify an equality predicate. The default is
The following facilities to control how results are reported is still under development. Comments welcome.
lisp-unit calls three listener functions to report test results and summary statistics. These functions can be rebound using the facilities below.
The test listener is called after each assertion form in a test is executed. The listener is passed
Two test listener functions are exported:
The error listener is called when an error occurs in running a test. The listener is passed
Three error listener functions are exported:
The summary listener is called after
The listener is passed:
Three summary listener functions are exported:
The three listeners are stored in the exported global variables
*summary-listener*. So one way to change listeners
For example, to show only package-level summary counts:
(let ((*summary-listener* 'show-package-summary)) (run-tests))
The above would still show failures and error messages. To hide those and just get the counts:
(let ((*error-listener* 'count-error) (*summary-listener* 'show-package-summary) (*test-listener* 'show-no-result)) (run-tests))
To show no individual test results and only package summaries with failures, we need to define a function that checks the number of failures.
(defun show-failing-package (name test-count pass-count error-count) (when (or (< pass-count test-count) (> error-count 0)) (show-summary name test-count pass-count error-count))) (let ((*error-listener* 'count-error) (*summary-listener* 'show-failing-package) (*test-listener* 'show-no-result)) (run-tests))
A simpler way to rebind listeners is with
The second example above could be done with:
(with-listeners (count-error show-package-summary show-no-result) (run-tests))
To use user-defined listeners, you
first say what listener variable they're for with
The third example above could be done with:
(defun show-failing-package (name test-count pass-count error-count) (when (or (< pass-count test-count) (> error-count 0)) (show-summary name test-count pass-count error-count))) (set-listener-variable 'show-failing-package '*summary-listener*) (with-listeners (count-error show-failing-package show-no-result) (run-tests))
Comments? Send me email.