At the lowest level testing the system requires that certain forms be evaluated and that certain post conditions are met: the value returned must satisfy a certain predicate, the form must (or must not) signal a certain condition, etc. In FiveAM these low level operations are called 'checks' and are defined using the various checking macros.
Checks are the basic operators for collecting results. Tests and test suites on the other hand allow grouping multiple checks into logic collections.
(eval-when (:compile-toplevel :load-toplevel :execute) (def-special-environment run-state () result-list current-test))
Every check produces a result object.
Class TEST-RESULT All checking macros will generate an object of type TEST-RESULT.
Class TEST-PASSED Class for successful checks.
(define-condition check-failure (error) ((reason :accessor reason :initarg :reason :initform "no reason given") (test-case :accessor test-case :initarg :test-case) (test-expr :accessor test-expr :initarg :test-expr)) (:documentation "Signaled when a check fails.") (:report (lambda (c stream) (format stream "The following check failed: ~S~%~A." (test-expr c) (reason c)))))
Class TEST-FAILURE Class for unsuccessful checks.
Class UNEXPECTED-TEST-FAILURE Represents the result of a test which neither passed nor failed, but signaled an error we couldn't deal with.
Class TEST-SKIPPED A test which was not run.
Function ADD-RESULT Create a TEST-RESULT object of type RESULT-TYPE passing it the initialize args MAKE-INSTANCE-ARGS and adds the resulting object to the list of test results.
Macro IS The DWIM checking operator.
Macro SKIP Generates a TEST-SKIPPED result.
Macro IS-TRUE Like IS this check generates a pass if CONDITION returns true and a failure if CONDITION returns false.
Macro IS-FALSE Generates a pass if CONDITION returns false, generates a failure otherwise.
Macro SIGNALS Generates a pass if BODY signals a condition of type CONDITION.
Macro FINISHES Generates a pass if BODY executes to normal completion.
Macro PASS Simply generate a PASS.
Macro FAIL Simply generate a FAIL.
FiveAM provides the ability to automatically generate a collection of random input data for a specific test and run a test multiple times.
Specification testing is done through the FOR-ALL macro. This macro will bind variables to random data and run a test body a certain number of times. Should the test body ever signal a failure we stop running and report what values of the variables caused the code to fail.
Variable *NUM-TRIALS* Number of times we attempt to run the body of the FOR-ALL test.
Variable *MAX-TRIALS* Number of total times we attempt to run the body of the FOR-ALL test including when the body is skipped due to failed guard conditions.
Since this is random testing we need some way of creating random data to feed to our code. Generators are regular functions whcih create this random data.
We provide a set of built-in generators.
(defgenerator gen-integer (&key (max (1+ most-positive-fixnum)) (min (1- most-negative-fixnum))) (+ min (random (1+ (- max min)))))
(defgenerator gen-long-float (&key (max (1+ most-positive-long-float)) (min (1- most-negative-long-float))) (+ min (random (1+ (- max min)))))
(defgenerator gen-character (&key (code (gen-integer :min 0 :max (1- char-code-limit))) (alphanumericp nil)) (if alphanumericp (code-char (funcall code)) (loop for char = (code-char (funcall code)) until (alphanumericp char) finally (return char))))
The trivial always-produce-the-same-thing generator is done using cl:constantly.
When running tests we often need to setup some kind of context (create dummy db connections, simulate an http request, etc.). Fixtures provide a way to conviently hide this context into a macro and allow the test to focus on testing.
NB: A FiveAM fixture is nothing more than a macro. Since the term 'fixture' is so common in testing frameworks we've provided a wrapper around defmacro for this purpose.
(deflookup-table fixture :documentation "Lookup table mapping fixture names to fixture objects.")
Macro DEF-FIXTURE Defines a fixture named NAME.
Macro WITH-FIXTURE Insert BODY into the fixture named FIXTURE-NAME.