Python Unit Testing Framework

Author: Steve Purcell, <stephen_purcell at yahoo dot com>
Project website: http://pyunit.sourceforge.net/

Contents

Overview

The Python unit testing framework, dubbed 'PyUnit' by convention, is a Python language version of JUnit, by smart cookies Kent Beck and Erich Gamma. JUnit is, in turn, a Java version of Kent's Smalltalk testing framework. Each is the de facto standard unit testing framework for its respective language.

This document explains the Python-specific aspects of the design and usage of PyUnit; for background information on the basic design of the framework the reader is referred to Kent's original paper, "Simple Smalltalk Testing: With Patterns".

PyUnit forms a part of the Python Standard Library as of Python version 2.1.

The following information assumes knowledge of Python, a language so easy that even I managed to learn it, and so addictive that I can't stop.

System requirements

PyUnit is designed to work with any standard Python, version 1.5.2 and higher.

PyUnit has been tested by the author on Linux (Redhat 6.0 and 6.1, Debian Potato) with Python 1.5.2, 2.0 and 2.1. It is also known to work on other Python platforms, including Windows and Mac. If any platform or Python version issues cause you trouble, please let me know.

For details of using PyUnit with JPython and Jython, please refer to the section 'Using PyUnit with JPython and Jython'.

Using PyUnit to write your own tests

Installation

The classes needed to write tests are to be found in the 'unittest' module. This module is part of the standard Python library for Python 2.1 and later. If you are using an older Python version, you should obtain the module from the separate PyUnit distribution.

To be able to use the module from your own code, simply ensure that the directory containing the file 'unittest.py' is in your Python search path. You can do this by setting the '$PYTHONPATH' environment variable, or by placing the file in a directory in your current Python search path, such as /usr/lib/python1.5/site-packages on Redhat Linux machines.

Note that you will have to do this before you can run the examples that are provided with PyUnit unless you copy 'unittest.py' into the examples directory.

An introduction to TestCases

The basic building blocks of unit testing are 'test cases' -- single scenarios that must be set up and checked for correctness. In PyUnit, test cases are represented by the TestCase class in the unittest module. To make your own test cases you must write subclasses of TestCase.

An instance of a TestCase class is an object that can completely run a single test method, together with optional set-up and tidy-up code.

The testing code of a TestCase instance should be entirely self contained, such that it can be run either in isolation or in arbitrary combination with any number of other test cases.

Creating a simple test case

The simplest test case subclass will simply override the runTest method in order to perform specific testing code:

        import unittest

        class DefaultWidgetSizeTestCase(unittest.TestCase):
            def runTest(self):
                widget = Widget("The widget")
                assert widget.size() == (50,50), 'incorrect default size'
    

Note that in order to test something, we just use the built-in 'assert' statement of Python. If the assertion fails when the test case runs, an AssertionError will be raised, and the testing framework will identify the test case as a 'failure'. Other exceptions that do not arise from explicit 'assert' checks are identified by the testing framework as 'errors'. (See also the section 'More about test conditions'.)

The way to run a test case will be described later. For now, note that to construct an instance of such a test case, we call its constructor without arguments:

        testCase = DefaultWidgetSizeTestCase()
    

Re-using set-up code: creating 'fixtures'

Now, such test cases can be numerous, and their set-up can be repetitive. In the above case, constructing a 'Widget' in each of 100 Widget test case subclasses would mean unsightly duplication.

Luckily, we can factor out such set-up code by implementing a hook method called setUp, which the testing framework will automatically call for us when we run the test:

        import unittest

        class SimpleWidgetTestCase(unittest.TestCase):
            def setUp(self):
                self.widget = Widget("The widget")

        class DefaultWidgetSizeTestCase(SimpleWidgetTestCase):
            def runTest(self):
                assert self.widget.size() == (50,50), 'incorrect default size'

        class WidgetResizeTestCase(SimpleWidgetTestCase):
            def runTest(self):
                self.widget.resize(100,150)
                assert self.widget.size() == (100,150), \
                       'wrong size after resize'
    

If the setUp method raises an exception while the test is running, the framework will consider the test to have suffered an error, and the runTest method will not be executed.

Similarly, we can provide a tearDown method that tidies up after the runTest method has been run:

        import unittest

        class SimpleWidgetTestCase(unittest.TestCase):
            def setUp(self):
                self.widget = Widget("The widget")
            def tearDown(self):
                self.widget.dispose()
                self.widget = None
    

If setUp succeeded, the tearDown method will be run regardless of whether or not runTest succeeded.

Such a working environment for the testing code is termed a fixture.

TestCase classes with several test methods

Often, many small test cases will use the same fixture. In this case, we would end up subclassing SimpleWidgetTestCase into many small one-method classes such as DefaultWidgetSizeTestCase. This is time-consuming and discouraging, so in the same vein as JUnit, PyUnit provides a simpler mechanism:

        import unittest

        class WidgetTestCase(unittest.TestCase):
            def setUp(self):
                self.widget = Widget("The widget")
            def tearDown(self):
                self.widget.dispose()
                self.widget = None
            def testDefaultSize(self):
                assert self.widget.size() == (50,50), 'incorrect default size'
            def testResize(self):
                self.widget.resize(100,150)
                assert self.widget.size() == (100,150), \
                       'wrong size after resize'
    

Here we have not provided a runTest method, but have instead provided two different test methods. Class instances will now each run one of the test methods, with self.widget created and destroyed separately for each instance. When creating an instance we must specify the test method it is to run. We do this by passing the method name in the constructor:

        defaultSizeTestCase = WidgetTestCase("testDefaultSize")
        resizeTestCase = WidgetTestCase("testResize")
    

Aggregating test cases into test suites

Test case instances are grouped together according to the features they test. PyUnit provides a mechanism for this: the 'test suite', represented by the class TestSuite in the unittest module:

        widgetTestSuite = unittest.TestSuite()
        widgetTestSuite.addTest(WidgetTestCase("testDefaultSize"))
        widgetTestSuite.addTest(WidgetTestCase("testResize"))
    

For the ease of running tests, as we will see later, it is a good idea to provide in each test module a 'callable' object that returns a pre-built test suite:

       def suite():
           suite = unittest.TestSuite()
           suite.addTest(WidgetTestCase("testDefaultSize"))
           suite.addTest(WidgetTestCase("testResize"))
           return suite
    

or even:

       class WidgetTestSuite(unittest.TestSuite):
           def __init__(self):
               unittest.TestSuite.__init__(self,map(WidgetTestCase,
                                                     ("testDefaultSize",
                                                      "testResize")))
    

(the latter is admittedly not for the faint-hearted)

Since it is a common pattern to create a TestCase subclass with many similarly named test functions, there is a convenience function called makeSuite provided in the unittest module that constructs a test suite that comprises all of the test cases in a test case class:-

       suite = unittest.makeSuite(WidgetTestCase,'test')
    

Note that when using the makeSuite function, the order in which the various test cases will be run by the test suite is the order determined by sorting the test function names using the cmp built-in function.

Nesting test suites

Often it is desirable to group suites of test cases together, so as to run tests for the whole system at once. This is easy, since TestSuites can be added to a TestSuite just as TestCases can be added to a TestSuite:-

       suite1 = module1.TheTestSuite()
       suite2 = module2.TheTestSuite()
       alltests = unittest.TestSuite((suite1, suite2))
    

An example of nesting test suites can be found in the file 'alltests.py', in the 'examples' subdirectory of the distribution package.

Where to place testing code

You can place the definitions of test cases and test suites in the same modules as the code they are to test (e.g. 'widget.py'), but there are several advantages to placing the test code in a separate module, such as 'widgettests.py':

Running tests interactively

Of course, the whole point of writing these tests is so that we can run them and find out if our software is working. The test framework uses 'TestRunner' classes to provide an environment in which your tests can execute. The most common TestRunner is TextTestRunner, which can run tests and report the results in textual form:

        runner = unittest.TextTestRunner()
        runner.run(widgetTestSuite)
    

By default, TextTestRunner prints its output to sys.stderr, but this can be changed by passing a different file-like object to its constructor.

Using TextTestRunner like this is an ideal way to run your tests interactively from within a Python interpreter session.

Running tests from the command line

The unittest module contains a function called main, which can be used to easily turn a test module into a script that will run the tests it contains. The main function uses the unittest.TestLoader class to automatically find and load test cases within the current module.

Therefore, if you name your test methods using the test* convention described earlier, you can place the following code at the bottom of your test module:

        if __name__ == "__main__":
            unittest.main()
    

Then, when you execute your test module from the command line, all of the tests contained therein will be run. Run the module with the '-h' option in order to see the options available.

To run arbitrary tests from the command-line, you can run the unittest module as a script, giving it the name of an test case or test suite:

        % python unittest.py widgettests.WidgetTestSuite
    

or

        % python unittest.py widgettests.makeWidgetTestSuite
    

You may also specify particular tests on the command-line. To run the TestCase subclass 'ListTestCase' in the module 'listtests' (see the 'examples' subdirectory of the distribution package) you can execute the command:

        % python unittest.py listtests.ListTestCase.testAppend
    

where 'testAppend' is the name of the test method that is to be run by the test case instance. To create and run ListTestCase instances for all the 'test*' methods in that class, you can run:

        % python unittest.py listtests.ListTestCase
    

The GUI test runner

There is a graphical front end that you can use in order to run your tests. It is written using Tkinter, the windowing toolkit shipped with Python on most platforms. It looks similar to the JUnit GUI.

To use the GUI test runner, simply run:

        % python unittestgui.py
    
or
        % python unittestgui.py widgettests.WidgetTestSuite
    

Note that here, again, the name entered for the test to be run should be the fully-qualified name of an object which returns a TestCase or TestSuite instance. It should not be the name of a pre-created test, since every test must be recreated each time it is run.

The use of the GUI test runner rather than the text test runner imposes a time overhead due to all those window updates; on my system, it takes an extra seven seconds per thousand tests. Your mileage may vary.

Documenting your tests

Usually, when a test is run its name is displayed by the TestRunner. This name is derived from the name of the test case class, and the name of the test method that the instance has been initialised to run.

However, if you supply a doc-string for a test method, the first line of that doc-string will be displayed when the test is run. This provides an easy mechanism for documenting your tests:

        class WidgetTestCase(unittest.TestCase):
            def testDefaultSize(self):
                """Check that widgets are created with correct default size"""
                assert self.widget.size() == (50,50), 'incorrect default size'
    

More about test conditions

I have suggested the use of Python's built-in assertion mechanism for checking conditions in test cases rather than a 'home-brewed' equivalent; assert is simple, concise and familiar.

Note, however, that if tests are run with Python's optimisation option turned on (generating '.pyo' bytecode files), the assert statements will be skipped, rendering the test cases quite useless.

For those who tend to work with Python's optimisation option enabled, I have included a method assert_ in the TestCase class. It is functionally equivalent to the assert built-in and will not be optimised away, but it is less convenient and results in less helpful error messages:

        def runTest(self):
            self.assert_(self.widget.size() == (100,100), "size is wrong")
    

For good measure I have also provided TestCase with failIf and failUnless methods:

        def runTest(self):
            self.failIf(self.widget.size() <> (100,100))
    

A test case method can also call fail in order to fail immediately:

        def runTest(self):
            ...
            if not hasattr(something, "blah"):
                self.fail("blah missing")
                # or just 'self.fail()'
    

Testing for equality

The most common type of assertion is an assertion of equality between two values or objects. If the assertion fails, the developer usually wants to see what the incorrect value actually was.

TestCase has a pair of methods called assertEqual and assertNotEqual for this purpose (with aliases failUnlessEqual and failIfEqual for those who prefer):

        def testSomething(self):
            self.widget.resize(100,100)
            self.assertEqual(self.widget.size, (100,100))
    

Testing for exceptions

Often a test will wish to check that an exception is raised in a certain set of circumstances. If the expected exception is not thrown, the test should fail. This is easy to do:

        def runTest(self):
            try:
                self.widget.resize(-1,-1)
            except ValueError:
                pass
            else:
                fail("expected a ValueError")
    

Usually, the source of the expected exception is a callable object; for that reason, TestCase has an assertRaises method. The first two arguments of the method are the expected exception as it would appear in an 'except' clause, and the callable object. The remaining arguments are those that should be passed to the callable object:

        def runTest(self):
            self.assertRaises(ValueError, self.widget.resize, -1, -1)
    

Re-using old test code with PyUnit

Some users will find that they have existing test code that they would like to run from PyUnit, without converting every old test function to a TestCase subclass.

For this reason, PyUnit provides a FunctionTestCase class. This subclass of TestCase can be used to wrap an existing test function. Set-up and tear-down functions can also optionally be wrapped.

Given the following test function:

        def testSomething():
            something = makeSomething()
            assert something.name is not None
            ...
    

one can create an equivalent test case instance as follows:

        testcase = unittest.FunctionTestCase(testSomething)
    

If there are additional set-up and tear-down methods that should be called as part of the test case's operation, they can also be provided:

        testcase = unittest.FunctionTestCase(testSomething,
                                             setUp=makeSomethingDB,
                                             tearDown=deleteSomethingDB)
    

Using PyUnit with JPython and Jython

Although PyUnit was written primarily for 'C' Python, it is possible to write PyUnit tests using Jython for your Java or Jython software. This can be preferable to trying to write JUnit tests using Jython. PyUnit also works correctly with Jython's predecessors, JPython 1.0 and 1.1.

Of course, Java does not have a TK GUI interface, so PyUnit's Tkinter-based GUI will not work with Jython. The text-only interface works just fine, however.

To do so, simply copy the standard C Python library module files 'traceback.py', 'linecache.py', 'stat.py' and 'getopt.py' to a location from which they can be imported by JPython. You can get these files from any distribution of C Python. (These guidelines are based on the standard library of C Python 1.5.x, and may not be correct for other Python versions.)

Now you can write your PyUnit tests exactly as you would with C Python.

Caveats

Assertions

See the caveats in the section "More about test conditions" above.

Memory use

When exceptions are raised during the running of a test suite, the resulting traceback objects are saved so that failure details can be formatted and printed at the end of the test run. Apart from simplicity, the benefit of this is that a future version of the GUI TestRunner will be able to allow post-mortem inspection of local and global variable values, which are stored with the traceback.

A possible side-effect is that when running test suites with very high failure rates, the memory usage of all these saved traceback objects could become a problem. Of course, if so many tests are failing, this memory overhead is the least of your problems.

Terms of use

You may freely use, alter and redistribute this software under the same liberal terms that apply to Python itself. All I ask is that my name, e-mail address and the project URL be retained in the source code and accompanying documentation, with a credit for me as the original author.

My motive for writing this software was to make a small contribution to the improvement of software quality in the world; I didn't bargain on getting any cash. (That's not to say that sponsorship would be unwelcome.)

Future plans

One key plan for the future is to integrate the TK GUI with the IDLE IDE. Volunteers are welcome!

Other than that, I have no great plans to extend the functionality of the module. I have kept PyUnit as simple as possible (but no simpler, hopefully!) because I believe that helper modules for such common testing tasks as log file comparison are better written by test writers than by myself.

Updates and community

News, updates and more are available at the project website.

Comments, suggestions and bug reports are welcome; simply e-mail me or, better still, join the very low-volume mailing list and post your comments there. There are a surprisingly large number of people already using PyUnit, and they all have wisdom to share.

Acknowledgements

Many thanks to Guido and his disciples for the Python language. In tribute, I have written the following haiku (or 'pyku', if you will):

Guido van Rossum
'Gawky Dutchman' gave birth to
Beautiful Python

I gratefully acknowledge the work of Kent Beck and Erich Gamma for their work on JUnit, which made the design of PyUnit a no-brainer.

Thanks also to Tim Voght; I discovered after I had implemented PyUnit that he had also implemented a 'pyunit' module as part of his 'PyWiki' WikiWikiWeb clone. He graciously gave me the go-ahead to submit my version to the community at large.

Many thanks to those who have written to me with suggestions and questions. I've tried to add appropriate credits in the CHANGES file in the download package.

Particular thanks to Jérôme Marant, who packaged PyUnit for Debian.

Related information


About the author

Steve Purcell is just a programmer at heart, working independently writing, applying and teaching Open Source software.

He recently acted as Technical Director for a Web/WAP start-up, but spends most of his time architecting and coding large Java systems whilst counterproductively urging his Java-skilled colleagues to take up Python instead.


Steve Purcell, <stephen_purcell at yahoo dot com>
$Id: PyUnit.html,v 1.20 2001/08/08 07:22:16 purcell Exp $