NAME
forked_harness.pl - run tests in parallel with Forks::Super
VERSION
0.97
SYNOPSIS
$ perl t/forked_harness.pl [options] [test-files]
|= test= 1/86; status: 0/0 time= 2.988s | t/00-use.t .. ok
...
|= test=86/86; status: 0/0 time=11.441s | t/43c-foo.t .. ok
All tests successful. 1 iterations.
Elapsed time: 73.919DESCRIPTION
The forked_harness.pl script runs a suite of unit tests in parallel using the Forks::Super framework. It can be used in any context where you might run the command make test. Aside from being able to finish running your test suite faster, this framework has many additional uses:
Intermittent failures
If you have one or more unit tests that only fail intermittently, forked_harness.pl can help you to run the test multiple times, isolating the test output of the failed tests.
# run intermittent.t 200 times, only output on failure
$ forked_harness.pl -x 200 -q t/intermittent.tStress testing
As forked_harness.pl will make use of more CPU while running your tests, it can expose problems with your tests or your code that only occur under high CPU loads.
Synchronization issues
forked_harness.pl can expose issues caused by running multiple instances of a test. For example, if your module or some of your test scripts need to write to a file with a hard-coded filename, multiple instances of the test might interfere with each other and cause the test to fail.
TEST FILES
There are several ways to specify which tests to run.
Command-line
The tests to run may be specified on the command-line.
$ forked_harness.pl t/00-load.t t/43-foo.t
$ forked_harness.pl t/*server*.tArguments with filesystem wildcard characters will be expanded to include all filenames that match the pattern, even on Windows.
$ENV{TEST_FILES}
If test files are not specified on the command-line and the environment variable TEST_FILES is set, forked_harness.pl will use that variable as the list of tests to run.
$ TEST_FILES=t-special/*.t forked_harness.plMakefile
If test files are not specified on the command-line or in the $ENV{TEST_FILES} variable, forked_harness.pl will look for a file called Makefile in the current directory and in the parent directory. If such a file is found, it is scanned for a line matching
TEST_FILES\s*=\s*(.*)and extracts the set of test files from that line.
If all of these methods to identify a set of test files fails, forked_harness.pl will fail with the message:
No test files specified, can't read defaults from Makefile!OPTIONS
forked_harness.pl recognizes several command-line options to fine tune its behavior. Any of these options may be used in combination, except as noted.
-a, --abort-on-fail
When this option is used and a test fails, no further tests are performed and a quit signal is sent to any currently running tests.
-C, --color
When this option is used, and a Term::ANSIColor module with at least version 3.00 can be located, the output of forked_harness.pl will be colorized. If a recent version of Term::ANSIColor can not be found, this option has no effect.
-d, --debug
When this option is used, forked_harness.pl reports what it is doing.
-e var=value, --env var=value
Each test script will inherit its environment from forked_harness.pl, which inherits its environment from the shell. You can pass additional environment variable settings to each test by specifying the -e option with key-value pairs one or more times.
# set $ENV{FOO}=BAR in each test
forked_harness.pl -e FOO=BAR ...
# unset $ENV{FOO} in each test
forked_harness.pl -e FOO= ...-h, --harness
When this option is used, each test is run inside the "test_harness" in ExtUtils::Command::MM function (the "runtests" in Test::Harness function on older perls) rather than run directly from a subshell of the forked_harness.pl script.
Running a test like this can have subtle effects on the test environment such as running the test program in a new session or under a different process group, which may or may not be important to the test.
--help
Outputs a list of command-line options with brief descriptions of what they do. After displaying help, causes the program to exit without running any tests.
-I directory, --include directory
Includes the specified directory in the test script's @INC path. May be included several times in order to specify several directories:
$ forked_harness.pl -I blib/foo -I blib/libIf no -I options are specified, the directories ./blib/lib and ./blib/arch are included by default.
-m numproc, --maxproc numproc
Runs up to numproc tests simultaneously. The default number of tests to run simultaneously is 2 * nc + 1, where nc is the number of logical CPU cores on your system.
You may consider using a smaller value if you have very CPU-intensive tests, or a larger value if your tests are not very intense (I don't know, maybe they spend most of their times in sleep statements).
You can also specify the number of tests to run simultaneously by setting the MAX_PROC environment variable before running forked_harness.pl:
$ forked_harness.pl --max-proc 15 [other-options] [test-files]
$ forked_harness.pl -m 15 [other-options] [test-files]
$ MAX_PROC=15 forked_harness.pl [other-options] [test-files]-O, --order
If specified, test results will be reported in their correct order. The default is to report results as they are available, so results for fast-running tests might appear before the results of slow-running tests that started earlier.
-p option, --popts option
Passes the specified option as a command-line option to the perl process running each test. Here are some examples of how this option might be useful:
Apply Carp::Always to get stack traces of any warnings in tests
$ forked_harness.pl -p -MCarp::Always [test-files]Run all tests in taint mode
$ forked_harness.pl -p -T [test-files]
$ forked_harness.pl -p -t [test-files]Run each test through a Devel:: module
$ forked_harness.pl -p -d:Trace::Fork [test-files]The -p option may be specified as many times as desired to pass more than one option to the perl program running the test.
-q, --quiet
Run in "quiet" mode. Suppresses output of successful tests except for a single summary line (see "OUTPUT"). This option hides the additional output produced when the -v (--verbose) option is used.
-qq, --really-quiet
Run in a very quiet mode. Suppresses all output of successful tests, and suppresses output of failed tests until all tests are completed. This option hides the additional output produced when the -v (--verbose) option is used.
-r times, --repeat times
Runs times iterations of the tests, aborting if there are test failures in any iteration. See also the -x option. The default is to run one iteration of the tests.
-s, --shuffle
If this option is included, the tests will be run in a random order.
-t timeout, --timeout timeout
Sets a timeout of timeout seconds on each test. If possible, tests will be terminated (and marked as failures) if they are still running when the timeout expires.
The default timeout is 150 seconds. Specify a timeout of 0 (-t 0, --timeout 0) to disable the timeout and give each test as long as it needs to complete.
The default timeout can also be overridden by setting the TEST_TIMEOUT environment variable.
-v, --verbose
When used with the -h (harness) options, runs each test in verbose mode, as if you had run the test suite with the command
make test TEST_VERBOSE=1The results of each individual test and anything else a test script writes to STDOUT will be output with this option.
-x times, --xrepeat times
Within each iteration (see the -r option), runs each test times times. The default is to run each test one time in each iteration.
-z, --socket
Instructs forked_harness.pl and Forks::Super to use socket based interprocess communication where possible, instead of file based IPC. File IPC should be more flexible and robust than socket IPC, so this option probably won't get you anything unless you are really really really low on disk space, or if you don't have write permission in the filesystem where you run your tests.
OTHER ENVIRONMENT AND CONFIGURATION
COLOR=some true value-
has the same effect as using the
-c(--color) flag. MAX_PROC=numprocs-
has the same effect as using the
-mnumprocs (--maxproc) option TEST_VERBOSE=some true value-
has the same effect as using the
-v(--verbose) flag. TEST_FILES=filenames-
specifies which tests to run, if they were not provided on the command line.
PROGRAM OUTPUT
At a minimum, forked_harness.pl produces a status line for each test that it runs:
|= test= 49/113; status: 1024/768 time=44.643s | t/41j-filehandles.t
A B C D E FA- the current test number-
The test results are reported when a test finishes. The reports may be out of order. If there is more than one iteration of testing planned (see the
-roption), this also reports the current iteration. B- total number of tests-
Total number of tests in this iteration. If there are multiple iterations (see the
-roption), then this also reports the number of iterations planned. C- status of this test-
Exit status of the test script. Anything other than a zero indicates a test failure. Normally this value is 256 times the number of failed tests, but it could have a different value to indicate that the test terminated abnormally.
D- previous status-
The highest exit code from the current test or any previous test.
E- test time-
Running time of the test, with millisecond resolution. When you get in the habit of running your unit tests in parallel, the longest running tests in your suite can be a bottleneck in your overall test time, and you may want to look for ways to break it into several smaller tests.
F- test name-
The test file that this line is reporting on.
Depending on which settings you are using, output from the test may follow the test's status line. Standard output and standard error will be separated, and all output from the test will appear at once.
EXIT STATUS
The exit status of forked_harness.pl will be the approximate number of test failures encountered. If there were more than 254 failures, however, the exit code will be 254.
This is the same behavior as ExtUtils::Command::MM test_harness function.
SEE ALSO
forked_harness.pl was originally developed as a proof-of-concent for the Forks::Super distribution, and there are two targets in the Makefile of Forks::Super that use forked_harness.pl:
# ------ fasttest: use Forks::Super to run Forks::Super tests in parallel
fasttest :: pure_all
$(FULLPERL) t/forked_harness.pl $(TEST_FILES) -h -q
# ------ stress test: run all tests in parallel 100 times
stresstest :: pure_all
$(FULLPERL) t/forked_harness.pl $(TEST_FILES) -r 20 -x 5 -s -qAUTHOR
Marty O'Brien, <mob@cpan.org>
LICENSE AND COPYRIGHT
Copyright (c) 2010-2018, Marty O'Brien.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.8 or, at your option, any later version of Perl 5 you may have available.
See http://dev.perl.org/licenses/ for more information.