Reloading Python modules

Problem

The GUI for PyUnit presented an unusual challenge; every time it is asked to run a test, it must ensure that the modules referenced by the test are completely reloaded. That way, the GUI can be left running while tests are altered and problems fixed.

It sounds deceptively simple, and proved to be rather fiddly. This page describes what I discovered.

Wrong solution 1: reload() all modules

The first obvious solution was to simply reload() all modules. This is easily accomplished using the following code snippet, or similar:

       for mod in sys.modules.values():
          reload(mod)
    

One could be clever, and only reload recent or non-built-in modules, but the idea remains the same. The reason this solution is wrong is illustrated by the following scenario. Consider two modules 'a' and 'b':

       ------- File 'a.py' ------
       value = 1

       ------- File 'b.py' ------
       from a import value
    

and some code that uses them:

       import a, b
       assert b.value == a.value == 1
    

Now imagine that we change 'a.py':

       ------- File 'a.py' ------
       value = 2
    

We want to reload all our modules so that they reflect the changes -- this is analogous to editing the code, then wanting to re-test it:

       reload(b)
       reload(a)
       assert a.value == 2
       assert b.value == 2
    

Here, the second assertion fails! This is due to the order in which the reload() calls were made. Grab the example code and try it for yourself.

Of course, the problem is that the modules are chained together by their imports.

Wrong solution 2: reload() all modules twice

"Aha!" one might say, "Why not reload those modules twice?" Indeed, in the above case this works fine, but it is not a general solution:

       ------- File 'a.py' ------
       value = 1
       ------- File 'b.py' ------
       from a import value
       ------- File 'c.py' ------
       from b import value
    

If these modules were imported, then 'a.value' changed in file 'a.py', the modules would need to be reloaded three times before 'c.value' would match 'a.value'.

Wrong solution 3: using the 'ihooks' package

At first glance, it appears that using a custom importer built using the 'ihooks' package should allow us to explicitly load every module afresh each time it is imported.

This could be achieved by simply not checking 'sys.modules' for pre-loaded modules. However, mutually referencing modules cause an endless loop if the 'sys.modules' shortcut is omitted.

Storing the modules in a list other than 'sys.modules' also caused problems.

Correct solution: a 'rollback importer'

Taking a step back, the situation is as follows:

1. A set of modules are loaded
2. The modules are used
3. The code is changed
4. The modules must be used again, but be freshly imported

The solution is to draw a line in the 'module sand' before loading and using the modules, then roll back to that point before re-running the code. This is accomplished, in PyUnit, by the following class:

class RollbackImporter:
    def __init__(self):
        "Creates an instance and installs as the global importer"
        self.previousModules = sys.modules.copy()
        self.realImport = __builtin__.__import__
        __builtin__.__import__ = self._import
        self.newModules = {}
        
    def _import(self, name, globals=None, locals=None, fromlist=[]):
        result = apply(self.realImport, (name, globals, locals, fromlist))
        self.newModules[name] = 1
        return result
        
    def uninstall(self):
        for modname in self.newModules.keys():
            if not self.previousModules.has_key(modname):
                # Force reload when modname next imported
                del(sys.modules[modname])
        __builtin__.__import__ = self.realImport
    

RollbackImporter instances install themselves as a proxy for the built-in __import__ function that lies behind the 'import' statement. Once installed, they note all imported modules, and when uninstalled, they delete those modules from the system module list; this ensures that the modules will be freshly loaded from their source code when next imported.

The rollback importer is used as follows in the PyUnit GUI (modified for clarity):

    def runClicked(self):
        if self.rollbackImporter:
            self.rollbackImporter.uninstall()
        self.rollbackImporter = RollbackImporter()
        self.loadAndExecuteTest()