NAME

run_tests.pl -- Run makepp regression tests

SYNOPSYS

run_tests.pl [-d] [-v] test1.test test2.test

If no arguments are specified, defaults to *.test.

DESCRIPTION

This script runs the specified tests and reports their result. With the -d option it only prints a dot for each successful test. A test that is skipped for a standard reason outputs a letter instead of a dot. The letters are l or m for build cache tests that were skipped because links don't work or MD5 is not available, s for a repository test skipped because symbolic links don't work or w for a Unix test skipped because you are on Windows. An x means the test can't be executed because that would require a Shell. If the test declares itself to not be relevant, that gives an r. Other reasons may be output as -.

With the -v option it also gives info about the used Perl version and system, handy when parallely running this on many setups, and the used time for the runner (and perl scripts it runs directly) on the one hand and for the makepp (and shell) child processes on the other hand.

A test is stored as a file with an extension of .test (very economic and -- with some care -- editable spar format), or .tar, .tar.bz2 or .tar.gz.

First a directory is created called tdir (called the test directory below). Then we cd to the directory, then extract the contents of the tar file. This means that the tar file ought to contain top-level files, i.e., it should contain ./Makeppfile, not tdir/Makeppfile.

A test may also be the name of an existing directory. In that case, no archive is unpacked and no cleanup is performed after the test.

The following files within this directory are important:

is_relevant.pl / is_relevant

If this file exists, it should be a perl script which return true (1) or shell script which returns true (0) if this test is relevant on this platform, and dies or false if the test is not relevant.

The first argument to this script is the full path of the makepp executable we are testing. The second argument is the current platform as seen by Perl. The environment variable PERL is the path to the perl executable we are supposed to use (which is not necessarily the one in the path).

makepp_test_script.pl / makepp_test_script

If this file exists, it should be a perl script or shell script which runs makepp after setting up whatever is necessary. If this script returns false (a non-zero value), then the test fails.

In a Perl script you can use the predefined function makepp() to run it with the correct path and wanted interpreter. It will die if makepp fails. You can also use the function wait_timestamp( file ... ), which will wait for both the local clock and the timestamp of newly created files to be at least a second later than the newest given file. You also have the function n_files, the first optional argument being a file name, where to write the count of built files, the second a sub that gets called for each log line so you can scan for messages. File::Copy's cp is also provided.

The first argument to this shell script is the full path of the makepp executable we are testing. The environment variable PERL is the path to the perl executable we are supposed to use (which is not necessarily the one in the path).

This script must be sufficiently generic to work in all test environments. For example:

• It must not assume that perl is in the path. Always use $PERL instead.

• It must work with the Bourne shell, i.e., it may contain no bash extensions.

• It must not use "echo -n" because that doesn't work on HP machines. But you should use &echo and other builtins for efficiency anyway.

If this file does not exist, then we simply execute the command $PERL makepp, so makepp builds all the default targets in the makefile.

makeppextra.pm

If present this module is loaded into perl before the script by the makepp function. See additional_tests/2003_11_14_timestamp_md5.test for an example of output redirection.

Makefile or Makeppfile

Obviously this is kind of important.

answers

This directory says what the result should be after running the test. Each file in the answers directory, or any of its subdirectories, is compared to a file of the same name in the test directory (or its corresponding subdirectory). The files must be exactly identical or the test fails.

Files in the main test directory do not have to exist in the answers subdirectory; if not, their contents are not compared.

There is one special file in the answers subdirectory: the file answers/n_files should contain two integers in ASCII format which are the number of files that makepp ought to build and that are expected to have failed. This is compared to the corresponding number of files that it actually built, extracted from the logfile .makepp/log.

cleanup_script.pl / cleanup_script

If this file exists, it should be a perl script or shell script that is executed when the test is done. This script is executed just before the test directory is deleted. No cleanup script is necessary if the test directory and all the byproducts of the test can be deleted with just rm -rf tdir. (This is usually the case, so most tests don't include a cleanup script.)