Acceptance Test Suite for PBS.
The acceptance tests is a suite of automated tests that testes most of the functionality in PBS. This document first gives an overview of the tests, and then describes the framework for the test suite and how to add additional test cases.
Overview of the Tests
The test suite can be divided in two sections, basic funtionality and rebuild correctness.
Basic functionality is tested with just one or a couple of functionality aspects at a time.
Rebuild correctness is that the correct set of nodes is rebuilt in a build. Different types of dependency graphs is tested, with some nodes modified and some unchanged in different combinations.
Here is a list of in which files the different types of tests resides.
Basic functionality
Build rules
t/Rules/*.pm
AddConfig, AddConfigTo
t/Misc/AddConfig.pm
PbsUse
t/Misc/PbsUse.pm
SubPbs files
t/Misc/Subpbs.pm
Node types
t/Misc/Nodetypes.pm
Source directories
t/Misc/SourceDirectory.pm
Targets
t/Misc/Targets.pm
Extra dependencies (changing the digest)
t/Misc/ExtraDependencies.pm
C dependency graph
t/Misc/CDependencyGraph/*.pm
Triggers
t/Misc/Triggers/*.pm
Warp
t/with_warp*.t
Command line flags
t/Misc/CommandLineFlags.pm
Error handling
t/ErrorHandling/*.pm
Rebuild correctness
t/Correctness/*.pm
Framework
The framework uses the modules Test::Class, Test::More, Test::Harness, and indirectly Test::Builder, for the infrastructure. Test::Class is a xUnit-style test framework that is used for organizing and executing tests. Test::More is used to expressing the testing conditions. Test::More uses Test::Builder as a backend. Test::Harness collects the results of the tests, reports them and presents statistics.
Additionally, the module Test::Cmd supplies the mechanics of setting up a testing environment for each test case, creating test files and running PBS and other programs.
Organization
First, there is a number of classes inheriting from Test::Class, each containing a collection of test cases that together test a specific aspect of PBS.
Second, there is a number of .t-files. Each .t-file runs all testes in all classes. This is done by using the module t::AllTests which enumerates all test classes. The difference between the .t-files is that they represents different configurations of PBS. Before they start running the tests they set the configuration to use. There is one exception from this. The .t-file t/run_many_times.t runs just a few of all the test cases, but repeats them multiple times.
Finally, there is the script pbs-test that runs all test cases in all configurations by running all the .t-files. This script accepts some switches that controls the amount of output shown during the tests. Run pbs-test --help to read about the switches.
This organization makes it possible to run the tests from Module::Build, CPan and others as part of an automated installation process.
In the individual test classes there is a number of methods, each constituting one test case. The methods express the testing conditions by using functions in Test::More. One such condition is the atomic unit of the test suite, one test. Tests fail and succeed in this unit, and the results and statistics presented by Test::Harness have this unit. A test case method is marked as such by using the attribute Test(num_tests), where num_tests is the number of tests in the test case.
Each test class can also have a startup, a setup, a teardown and a shutdown method, marked with corresponding attributes (see Test::Class. The startup method is executed before all the tests cases in the class. The shutdown method is executed after all the tests cases in the class. The setup method is executed before each test case and the teardown is executed after each test case. The most used method is the setup method, where the testing environment for each test case is setup.
Tests can be marked as todo tests. A todo test is expected to fail, and if it would succeed, it would be reported. This can be used for marking things that does not work for the moment, but will be fixed later on. If one just comments out the failing test case, to make the tests pass, there is a big chance one forgets the failing test. Look at the documentation for Test::More to read more about todo tests.
Testing Utilities
The class t::PBS supplies some functionality that is used by most of the test cases. It inherits from Test::Cmd and customizes that functionality to PBS. It also adds functionality not present in Test::Cmd. t::PBS creates a temporary directory that can be used to host all test files created during the execution of a test case. After the test case is finished, the directory is removed. The class can also run arbitrary commands and capturing the output.
Code Coverage Metrics
The test suite has support for generating code coverage metrics using the module Devel::Cover. That can be achived for instance using Module::Build and issuing the command ./Build testcover.
Internally, that works as follows. If the tests are started with the environment variable HARNESS_PERL_SWITCHES set to '-MDevel::Cover', that is, the tests are running under Devel::Cover and Test::Harness, then the sub Perl processes is started with Devel::Cover too, with the coverage data base path set to ./cover_db. Then coverage data will be collected for PBS too.
Files
Here is a summary of the files in the test suite.
documentation.pod - The source of this document.
pbs-test - A script that runs the whole test suite.
t/AllTests.pm - Module which enumerates all test classes.
t/PBS.pm - Utility class used by the test classes.
t/TestClassTemplate.pm - Template for a test class.
t/*.t - Scripts that run test cases under different configurations.
t/*/*.pm - Test classes.
Adding a test case
If the test case should be added to an existing test class, just add a method in the class.
If the test case should be added to a new class, there are more steps involved. Here is the recommended way of doing this:
- 1. Copy the template class file t/TestClassTemplate.pm to a new file.
- 2. Change the package header to reflect the new name.
- 3. Change the name in the setup method and do more customizations if needed.
- 4. Add the new class to the list in t/AllTests.pm.
- 5. Write the test case in a test method in the class.
Look at existing test cases, for instance t/Misc/ExtraDependencies.pm to see common techniques for testing PBS, and look at the documentation for t::PBS to see which utilities already exist.