NAME

TAP::Harness::ReportByDescription - Report TAP output by test file description rather than test file name

VERSION

0.01

SYNOPSIS

use TAP::Harness::ReportByDescription;
my $harness = TAP::Harness::ReportByDescription->new();
$harness->aggregate_tests($aggregator, @tests);

DESCRIPTION

This package subclasses TAP::Harness for the purpose of enabling a user to report the TAP output for a test file by a user-provided description rather than by the name of the test file itself.

Why would you want to do this? Three reasons come to mind.

• One Master Summary Rather Than Summaries for Individual Subharnesses

  Suppose that you had a make testing target that is in essence nothing more than a sequential run of several smaller testing targets, each of which is a separate invocation of a test harness.

  make fulltest : test_prep \
      compilers \
      src \
      oo \
      codingstd \
      examples

  Other things being equal, you would get a Summary at the end of each of the five targets or subharnesses. Under some circumstances, you might prefer to get a single master Summary at the end of the overall program.

• Multiple Runs of Same Tests in Different Environments

  Suppose that you had a set of tests that you wanted to run several times, each time in a slightly different environment. You could write a program which executes multiple runs, writing a summary after each run and then modifying the environment for the next run.

  perl t/harness --gc-debug --runcore=bounds
  perl t/harness --gc-debug --runcore=fast
  perl t/harness --gc-debug --run-pbc

  As the TAP output flowed by, you would see three instances of each test:

  t/pmc/arrayiterator.t ............................ ok
  # ...
  t/pmc/arrayiterator.t ............................ ok
  # ...
  t/pmc/arrayiterator.t ............................ ok

  ... but you would not be able to tell from the test file's report itself which harness it was a part of. Under certain circumstances it would be nice to be able to differentiate among the different runs:

  bounds__t/pmc/arrayiterator.t .................... ok
  # ...
  fast__t/pmc/arrayiterator.t ...................... ok
  # ...
  pbc__t/pmc/arrayiterator.t ....................... ok

  Here you're providing a description of each run of each test which provides an observer with more information.

• Preparation of a Test Harness Archive

  The ability to provide a specific description for a different run of the same test becomes crucial when preparing a test harness archive. Currently, CPAN distribution Test::Harness::Archive stores the TAP for a particular test file in a file with the name of the test file itself. If you do multiple runs of the same file in different environments, a later run of a test will overwrite the TAP file from an earlier run. You would therefore only be able to include the TAP from the last subharness in an archive. That would impede you from sharing the full results of testing via a smoke-test aggregator such as Smolder.

In short, we need (a) a way to run multiple harnesses as if they were one, (b) run the same tests through multiple harnesses and be able to quickly identify which harness we were running it through, and (c) store multiple versions of a file's TAP output in a test harness archive.

Need (a) can actually be fulfilled with existing TAP::Parser::Aggregator functionality. Let's build on that to meet needs (b) and (c). To do that we need one package to subclass TAP::Harness and one to subclass TAP::Harness::Archive. TAP::Harness::ReportByDescription and TAP::Harness::Archive::MultiplesHarnesses are these packages.

METHODS

new()

Inherited from TAP::Harness.

aggregate_tests()

Replicated, along with methods called internally from this method, from TAP::Harness. The only change occurs in an internal method _get_parser_args(), which now assigns the individual test's filename to one variable and a user-provided description to a second variable.

my $test_prog = $job->filename;
my $spool_prog = $job->description;

It is the latter variable which will appear on the console and in a test archive. Since this occurs within an internal method, the user need make no change in how aggregate_tests() is called.

EXAMPLE

See TAP::Harness::Archive::MultipleHarnesses::runtests().

AUTHOR

99% of the code in this module comes from TAP::Harness, written by Andy Armstrong and generations of Perl QA hackers. Documentation and the one small code tweak needed were written by James E Keenan.

LICENSE

This is free software and is released under the same terms as Perl itself.