-
Notifications
You must be signed in to change notification settings - Fork 5
PracticalUnitTestingGuide
With a clear rationale behind the desire for making use of automated testing, and in particular the benefits to obtain by way of unit testing, this page discusses the practicalities of writing such tests in the MiG project.
This attempts to be a concrete look at a number of basic facilities that have been implemented in the project to allow tests to be be written and a series of supporting pieces that provide a groundwork for comprehensive coverage.
The easiest way to execute the entire test suite is to invoke it via the top-level Makefile. Invoking make without a target will output a description of the basic targets, the two most most important are the following:
- unittest
- test
The same test code is executed by both targets, but the difference lies in the environment in which those tests will be run.
As a software developer, and when making changes to the system, one will most often make use of the unittest target: make unittest executes the tests in a virtual environment built atop the host system version of Python. This is fast and is intended for rapid iteration.
The necessary dependencies will be installed in a virtual environment automatically and no additional extra work is required to immediately validate the checkout.
For those wishing to run all available tests as a validation and to provide a level of assurance of all code currently covered, we urge use of the test target. Running make test will run the test suite inside a container of the minimum supported Python version, and thus represents validation of the code in a form closer to active production use.
We attempt to follow arrange-act-assert when writing test cases. That is, a tests should first arrange the conditions/environment that it needs to operate, perform a single clear activity and one or more assertions of statements that establish whether the activity succeeded and/or was performed correctly.
The MiG codebase is shared by a number of information management platforms each of which are a particular configuration of its underlying facilities. For this reason almost every interaction with logic in the application must be supplied a configuration - most often in the form of an argument. Thus there must be a way to easily obtain such a Configuration object.
see: provisioning configuration
Another important comment is that the history of the project and its purpose means a heavy use of files, not just those stored by users but also many of which contain persisted metadata in the filesystem. This means that tests can require the construction and use of many different paths, and left unchecked those details can quickly swamp both act but critically also the assertions which document resultant state.
see: filesystem isolation & helper abstractions
While a large amount of the MiG code is written as library calls and is thus very amenable to simple unit testing, there is a substantial of other code that is either executed in the background, as scripts, or is web facing and/or related code handling requests and generating responses for HTTP clients.
see: helpers abstractions
The MigTestCase is a subclass of the the standard unittest package test case definition, which means MiG tests are fully executable in any cpython installation.
This class is the what the individual files containing test functions for the logic under test will themselves subclass.
Test cases which define a _provide_configuration() will automatically have a valid configuration accessible via self.configuration which can be passed into any function that requires it.
In order to allow the system and make it able to run locally a basic local configuration is created on first invocation of the Makefile, and additionally a specific test configuration.
Return "testconfig" will expose this configuration via self.configuration which contains paths that are correctly configured set both when the tests are run locally via make unittest and within containers with make test.
TODO: explain fake configuration and its use for customised configs
With many operations eventually causing changes to files in the filesystem it is important when ensuring that any such changes are cleaned up between tests to ensure they cannot affect each other and results are accurate.
The TEST_OUTPUT_DIR constant is exported by the central tests.support library alongside the MigTestCase itself which allows creating paths within a directory that is automatically cleaned between tests runs.
The test configuration is specifically defined with the paths it accesses purposely placed within this cleaned space which corresponds with the TEST_OUTPUT_DIR constant and thus any path derived from it. Temporary file paths can be created within this space via the temppath() function.
Much like the temppath() function described above, there a number of other such helpers which one must opt-into based on the requirements of the tests.
Within the codebase there are a series of *supp.py files which are named subpackages of tests.support. These expose functions to be use as part of the arrange phase of a test, thus they establish some particular state or environment that allows the code to be tested. They also provide mixin classes with additional assertions.
As more and more code has gradually come under test a few challenges have been encountered, and in many cases that led to the introduction of facilities or served as strong influence on them.
There are however some things that must simply be tackled when tests are being introduced - often due to the code being brought under test not being designed to do so - and thus