The LIsp Framework for Testing (LIFT) is a unit and system test tool for LISP. Though inspired by SUnit and JUnit, it's built with Lisp in mind. In LIFT, testcases are organized into hierarchical testsuites each of which can have its own fixture . When run, a testcase can succeed, fail, or error. LIFT supports randomized testing, benchmarking, profiling, and reporting.
LIFT supports interactive testing so imagine that we type each of the following forms into a file and evaluate them as we go.
(in-package #:common-lisp-user)
(use-package :lift) First, we define an empty testsuite. deftestsuite is like defclass so here we define a testsuite with no super-testsuites and no slots.
> (deftestsuite lift-examples-1 () ())
==> #<lift-examples-1: no tests defined> Add a test-case to our new suite. Since we don't specify a testsuite or a test name, LIFT will add this to the most recently defined testsuite and name it for us.
> (addtest (ensure-same (+ 1 1) 2))
==> #<Test passed> Add another test using ensure-error Here we specify the testsuite and the name.
> (addtest (lift-examples-1) ; the testsuite name
div-by-zero ; the testcase name
(ensure-error (let ((x 0)) (/ x))))
==> #<Test passed> Though it works, ensure-error is a bit heavy-handed in this case. We can use ensure-condition to check that we get exactly the right kind of error.
> (addtest (lift-examples-1)
div-by-zero
(ensure-condition division-by-zero
(let ((x 0)) (/ x))))
==> #<Test passed> Notice that because we named the testcase div-by-zero, LIFT will replace the previous definition with this one. If you don't name your tests, LIFT cannot distinguish between correcting an already defined test and creating a new one.
Now, let's us run-tests to run all our tests. Unless you tell it otherwise, run-tests runs all the test-cases of the most recently touched testsuite 1 . Here, thats lift-example-1.
> (run-tests)
==> #<Results for lift-examples-1 [2 Successful tests]> As you saw above, if you don't supply a test-case name, LIFT will give it one. This works for quick interactive testing but makes it hard to find a problem when running regression tests. It's a much better practice to give every test-case a name -- it also makes the testsuite self documenting.
Here is a test-case that fails because floating point math isn't exact.
> (addtest (lift-examples-1)
floating-point-math
(ensure-same (+ 1.23 1.456) 2.686))
==> #<Test failed> Hmmm, what happened? Lift returns a test-result object so we can look at it to understand what went wrong. Let's describe it:
> (describe *)
Test Report for lift-examples-1: 1 test run, 1 Failure.
Failure: lift-examples-1 : floating-point-math
Condition: Ensure-same: 2.6859999 is not equal to 2.686
Code : ((ensure-same (+ 1.23 1.456) 2.686)) We try again using the function almost= for the test of ensure-same
> (addtest (lift-examples-1)
floating-point-math
(ensure-same (+ 1.23 1.456) 2.686 :test 'almost=))
==> #<Error during testing> Whoopts, we forgot to write almost=! Here's a simple (though not very efficient) version
> (defun almost= (a b)
(< (abs (- a b)) 0.000001))
==> almost= Like run-tests, run-test runs the most recently touched test-case.
> (run-test)
==> #<lift-examples-1.lift-examples-1 passed> The examples above cover most of LIFT's basics:
In what follows, we'll explore LIFT in more depth by looking at test hierarchies and fixtures, randomized testing, and using LIFT for benchmarking and profiling.
The deftestsuite macro defines or redefines a testsuite. Testsuites are CLOS classes and deftestsuite looks a lot like defclass.
(deftestsuite name (supersuite*)
(slotspec*)
options*) The list of supersuites lets you organize tests into a hierarchy. This can be useful both to share fixtures (i.e., setup and teardown code) and to organize your testing: different parts of the hierarchy can test different parts of your software. The slotspecs are similar to slotspecs in defclass but with a twist: deftestsuite automatically adds an initarg and accessor for each spec 2 . You can specify an initial value using a pair rather than needing to specify an initform . Finally, you'll also see below that slot values are immediately available with the body of a test method . These two features make writing tests very simple.
> (deftestsuite test-slots ()
((a 1) (b 2) c)
(:setup (setf c (+ a b)))
(:test ((ensure-same (+ a b) c))))
Start: test-slots
#<Results for test-slots [1 Successful test]> The example above also shows that you can define tests directly in the deftestsuite form. This is really handy for unit testing where you don't want the boilerplate to get in the way of the tests! Here is another, more complex example:
> (deftestsuite test-leap-year-p ()
()
;; Use :tests to define a list of tests
(:tests
((ensure (leap-year-p 1904)))
;; we give this one a name
(div-by-four (ensure (leap-year-p 2000)))
((ensure (leap-year-p 1996))))
;; use :test to define one test at a time
(:test ((ensure-null (leap-year-p 1900))))
(:test ((ensure-null (leap-year-p 1997)))))
;; let's see what we've done
> (print-tests :start-at 'test-leap-year-p)
test-leap-year-p (5)
TEST-1
div-by-four
TEST-3
TEST-4
TEST-5 So far, our tests have not required any setup or teardown. Let's next look at at a few tests that do. The first example is from the ASDF-Install testsuite. It uses its fixtures setup to make sure that the working directory is empty (so that it is ensured of installing into a clean system). 3
(deftestsuite test-asdf-install-basic-installation (test-asdf-install)
()
(:dynamic-variables
(*verify-gpg-signatures* t))
(:setup
(delete-directory-and-files *working-directory*
:if-does-not-exist :ignore))) This next testsuite is from Log5. Though the details aren't important, you can be assured that LIFT will run the setup before every test-case and the teardown after every test-case (even if there is an error).
(deftestsuite test-stream-sender-with-stream (test-stream-sender)
(sender-name
string-stream
(sender nil))
(:setup
(setf sender-name (gensym)
string-stream (make-string-output-stream)))
(:teardown (stop-sender-fn sender-name :warn-if-not-found-p nil))
:equality-test #'string-equal)
We've already seen two other clauses that deftestsuite supports (:dynamic-variables and :equality-test). Here is the complete list:
Many of these are self-explanatory. We'll discuss :dynamic-variables, :equality-test, :function, :run-setup and :timeout here and look at :random-instance below when we talk about random-testing.
It is often the case that you'll want some dynamic variable bound around the body of all of your tests. This is hard to do because LIFT doesn't expose its inner mechanisms for easy access. 4 The :dynamic-variables clause lets you specify a list of variables and bindings that LIFT will setup for each testcase.
This is used to specify the default equality-test used by ensure-same for test-cases in this suite and any suites that inherit from it. Though you can use the special variable emphasis to set test, it usually better to exercise control at the testsuite level. This is especially handy when, for example, you are testing numeric functions and want to avoid having to specify the test for every ensure-same.
Let the Common Lisp forms flet, labels, and macrolet, deftestsuite's function clause lets you define functions that are local to a particular testsuite (and its descendants). There are two good reasons to use :function: it provides good internal documentation and structure and you can use the testsuite's local variables without without any fuss or bother. Here is an example:
(deftestsuite test-size (api-tests)
(last-count db)
(:function
(check-size (expected)
(ensure (>= (size) last-count))
(setf last-count (size))
(ensure-same (size) (count-slowly db))
(ensure-same (size) expected)))
(:setup
(setf db (open-data "bar" :if-exists :supersede))) The check-size function will not conflict with any other check-size functions (from other tests or any of Lisp's other namespaces). Secondly, the references to last-count and db will automatically refer to the testsuite's variables.
LIFT's usual behavior is to run a testsuite's setup and teardown code around every single test-case. This provides the best isolation and makes it easy to think about a test-case by itself. If test setup takes a long time or if you want to break a complex test into a number of stages, then LIFT's usual behavior will just get in the way. The run-setup clause lets you control when setup (and teardown) occur. It can take on one of the following values:
setup and teardown around every testcasesetup for the first test-case of a testsuite and run teardown after the last test-case.Things go wrong (that is, after all, part of why we write tests!). The timeout clause lets you tell LIFT that if test-case hasn't completed within a certain number of seconds, then you want LIFT to complete the test with an error.
To be written.
To be written.
To be written.
Creates a testsuite named testsuite-name and, optionally, the code required for test setup, test tear-down and the actual test-cases. A testsuite is a collection of test-cases and other testsuites.
Test suites can have multiple superclasses (just like the classes that they are). Usually, these will be other test classes and the class hierarchy becomes the test case hierarchy. If necessary, however, non-testsuite classes can also be used as superclasses.
Slots are specified as in defclass with the following additions:
my-slot, then the initarg will be :my-slot and the accessors will be my-slot and (setf my-slot).
:initform for the slot. I.e., if you have
(deftestsuite my-test ()
((my-slot 23))) then my-slot will be initialized to 23 during test setup.
Test options are one of :setup, :teardown, :test, :tests, :documentation, :export-p, :dynamic-variables, :export-slots, :function, :categories, :run-setup, or :equality-test.
:categories - a list of symbols. Categories allow you to groups tests into clusters outside of the basic hierarchy. This provides finer grained control on selecting which tests to run. May be specified multiple times.
:documentation - a string specifying any documentation for the test. Should only be specified once.
:dynamic-variables - a list of atoms or pairs of the form (name value). These specify any special variables that should be bound in a let around the body of the test. The name should be symbol designating a special variable. The value (if supplied) will be bound to the variable. If the value is not supplied, the variable will be bound to nil. Should only be specified once.
:equality-test - the name of the function to be used by default in calls to ensure-same and ensure-different. Should only be supplied once.
:export-p - If true, the testsuite name will be exported from the current package. Should only be specified once.
:export-slots - if true, any slots specified in the test suite will be exported from the current package. Should only be specified once.
:function - creates a locally accessible function for this test suite. May be specified multiple times.
:run-setup - specify when to run the setup code for this test suite. Allowed values are
:run-setup is handy when a testsuite has a time consuming setup phase that you do not want to repeat for every test.
:setup - a list of forms to be evaluated before each test case is run. Should only be specified once.
:teardown - a list of forms to be evaluated after each test case is run. Should only be specified once.
:test - Define a single test case. Can be specified multiple times.
The following macros can be used outside of LIFT where they will function very much like assert. When used in the body of an addtest or deftestsuite form, however, they will record test failures instead of signaling one themselves.
5
If ensure's predicate evaluates to false, then it will generate a test failure. You can use the report and arguments keyword parameters to customize the report generated in test results. For example:
(ensure (= 23 12)
:report "I hope ~a does not = ~a"
:arguments (12 23)) will generate a message like
Warning: Ensure failed: (= 23 12) (I hope 12 does not = 23) predicate evaluates to true, then it will generate a test failure. You can use the report and arguments keyword parameters to customize the report generated in test results. See ensure for more details.
report as a format string and arguments as arguments to that string (if report and arguments are supplied). If ensure-same is used within a test, a test failure is generated instead of a warning
arguments as arguments to that string (if report and arguments are supplied). If ensure-different is used within a test, a test failure is generated instead of a warning
Signal a test-failure if body does not signal condition.
If condition is an atom, then non-error conditions will not cause a failure.
condition may also be a list of the form
(condition &key catch-all-conditions? report arguments name validate) If this form is used then the values are uses as follows:
report and arguments are used to display additional information when the ensure fails.
`catch-all-conditions? - if true, then the signaling of any other condition will cause a test failure.
validate - if supplied, this will be evaluated when the condition is signaled with the condition bound to the variable condtion (unless name is used to change this). validate can be used to ensure additional constaints on the condition.
Many of the variables below are used as the default values when calling run-test or run-tests or when interactively defining new tests and testsuites.
If bound to a pathname or stream, then a summary of test information will be written to it for later processing. It can be set to:
nil - generate no outputt - send output to a pathname constructed from the name of the system being tested (this only works if ASDF is being used to test the system). As an example of the last case, if LIFT is testing a system named ...
*print-length* except that it can also take on the value :follow-print. In this case, it will be set to the value of *print-length*.
*print-level* except that it can also take on the value :follow-print. In this case, it will be set to whatever *print-level* is.
Call fn with each suite name starting at start-at
fn should be a function of two arguments. It will called with a testsuite name and the level of the suite in the class hierarchy.
Search for a testsuite named suite.
The search is conducted across all packages so suite can be a symbol in any package. I.e., find-testsuite looks for testsuite classes whose symbol-name is string= to suite. If errorp is true, then find-testsuite can raise two possible errors:
testsuite-ambiguous will be raised. testsuite-not-defined will be raised. The default for errorp is nil.
thing is a testsuite. Thing can be a symbol naming a suite, a subclass of test-mixin or an instance of a test suite. Returns nil if thing is not a testsuite and the symbol naming the suite if it is.
ensure variants like ensure-random-cases. ↩
Copyright (c) 2001 - 2012 Gary Warren King (gwking@metabang.com) LIFT has an MIT style license Last updated 2 September 2012 at 09:56