/ 
Test-Group-0.20
River stage one • 7 direct dependents • 8 total dependents
/ Test::Group

NAME

Test::Group - Group together related tests in a test suite

VERSION

Test::Group version 0.20

SYNOPSIS

Basics:

use Test::More no_plan => 1;
use Test::Group;

test "hammering the server" => sub {
    ok(I_can_connect);
    for(1..1000) {
       ok(I_can_make_a_request);
    }
}; # Don't forget the semicolon here!

Failed subtests are displayed before the result of the test they belong to. For instance,

use Test::More no_plan => 1;
use Test::Group;

test "this test group will fail", sub {
    ok 1, "sub test blah";
    is "foo", "bar", "I so wish they were the same...";
    ok 1;
    like   "blah blah blah", qr/bla/;
};

produces something like this:

#   Failed test 'I so wish they were the same...'
#   in test.pl at line 6.
#          got: 'foo'
#     expected: 'bar'
not ok 1 - this test group will fail
#   Failed test 'this test group will fail'
#   in test.pl at line 9.
1..1

Exceptions in tests are not fatal:

test "this test will fail but the suite will proceed", sub {
    pass;
    die;
};

Test::More style TODO support:

test "a test with TODO in the name is marked TODO" => sub {
      pass("this part is done");
      fail("but I'm not finished with this one yet");
};

{
  local $TODO = "Test::More's good old method also works";
  test "this test is not finished yet" => sub {
      pass;
      fail;
  };
};

Misc:

# Don't catch exceptions raised in test groups later on
Test::Group->dont_catch_exceptions;

# log caught exceptions in /tmp/log
Test::Group->logfile("/tmp/log");

# skip the next group of test
skip_next_test "network not available" if (! Network->available());
test "bla", sub {
    my $ftp = Net::FTP->new("some.host.name");
    # ...
};

begin_skipping_tests "reason";

test "this test will not run" => sub {
    # ...
};

end_skipping_tests;

# from now on, skip all tests whose names do not match /bla/
test_only qr/bla/;

DESCRIPTION

Fed up with counting tests to discover what went wrong in your last test run? Tired of squinting at your test source to find out where on earth the faulty test predicate is called, and what it is supposed to check for? Then this module is for you!

Test::Group allows for grouping together related tests in a standard Test::More-style script. (If you are not already familiar with Test::More, now would be the time to go take a look.) Test::Group provides a bunch of maintainability and scalability advantages to large test suites:

  • related tests can be grouped and given a name. The intent of the test author is therefore made explicit with much less effort than would be needed to name all the individual tests;

  • the test output is much shorter and more readable: only failed subtests show a diagnostic, while test groups with no problems inside produce a single friendly ok line;

  • no more tedious test counting: running an arbitrarily large or variable number of tests (e.g. in loops) is now hassle-free and doesn't clutter the test output.

Authors of Test::* modules may also find Test::Group of interest, because it allows for composing several Test::More predicates into a single one (see "Reflexivity").

FEATURES

Blocking Exceptions

By default, calls to "die" in perlfunc and other exceptions from within a test group cause it to fail and terminates execution of the group, but does not terminate whole script. This relieves the programmer from having to worry about code that may throw in tests.

This behavior can be disabled totally using "dont_catch_exceptions". Exceptions can also be trapped as usual using "eval" in perlfunc or otherwise from inside a group, in which case the test code of course has full control on what to do next (this is how one should test error management, by the way).

When Test::Group is set to block errors (the default setting, see also "catch_exceptions"), the error messages are displayed as part of the test name, which some may not find very readable. Therefore, one can use a "logfile" instead.

Skipping Groups

Test::Group can skip single test groups or a range of them (consecutive or matched by a regex), which helps shortening the debug cycle even more in test-driven programming. When a test group is skipped, the code within it is simply not executed, and the test is marked as skipped wrt Test::Builder. See "skip_next_test", "skip_next_tests", "begin_skipping_tests", "end_skipping_tests" and "test_only" for details.

Reflexivity

Test groups integrate with Test::Builder by acting as a single big test; therefore, Test::Group is fully reflexive. A particularly attractive consequence is that constructing new Test::More predicates is straightforward with Test::Group. For example,

use Test::Builder;
use Test::Group;

sub foobar_ok {
    my ($text, $name) = @_;
    $name ||= "foobar_ok";
    local $Test::Builder::Level = $Test::Builder::Level + 1;
    test $name => sub {
       local $Test::Group::InPredicate = 1;
       like($text, qr/foo/, "foo ok");
       like($text, qr/bar/, "bar ok");
    };
}

defines a new test predicate foobar_ok that will DWIM regardless of the caller's testing style: for "classical" Test::Simple or Test::More users, foobar_ok will act as just another *_ok predicate (in particular, it always counts for a single test, honors "TODO: BLOCK" in Test::More constructs, etc); and of course, users of Test::Group can freely call foobar_ok from within a group.

Adding 1 to $Test::Builder::Level causes the location of the call to foobar_ok() to be shown if a test fails, see Test::Builder.

Setting $Test::Group::InPredicate to a true value prevents the location of individual failing tests within test groups from being shown.

TODO Tests

As shown in "SYNOPSIS", Test::More's concept of TODO tests is supported by Test::Group: a group is in TODO state if the $TODO variable is set by the time it starts, or if the test name contains the word TODO. Note, however, that setting $TODO from inside the test group (that is, after the group starts) will not do what you mean:

test "something" => sub {
    local $TODO = "this test does not work yet";
    pass;                                         # GOTCHA!
    fail;
};

Here pass is an unexpected success, and therefore the whole test group will report a TODO success despite the test not actually being a success (that is, it would also be defective if one were to comment out the local $TODO line). This semantics, on the other hand, DWIMs for marking a portion of the test group as TODO:

test "something" => sub {
    pass;
    {
       local $TODO = "this part does not work yet";
       fail;
    }
};

Finally, there is a subtle gotcha to be aware of when setting $TODO outside a test group (that's the second one, so maybe you should not do that to begin with). In this case, the value of $TODO is set to undef inside the group. In other words, this test (similar to the one to be found in "SYNOPSIS") will succeed as expected:

{
  local $TODO = "not quite done yet";
  test "foo" => sub {
      fail;
      pass;              # NOT an unexpected success, as
                         # this is simply a subtest of the whole
                         # test "foo", which will fail.
  };
}

OUTPUT FORMAT

As seen briefly in "SYNOPSIS", only top-level test groups (and toplevel Test::More tests if any) produce a single "ok" or "not ok" summary line. Failed sub-tests produce non-scoring comment messages (prefixed with "#"); successful sub-tests are silent. This is different from, and predates, the "subtest" in Test::More functionality.

However, if you enable the experimental "use_subtest" feature then Test::Group will use the same underlying mechanism as "subtest" in Test::More and produce very similar output.

PLUGIN INTERFACE

A simple plugin interface allows module authors to write extensions to Test::Group. See Test::Group::Extending for details.

The following extensions are distributed with Test::Group:

Test::Group::Plan, Test::Group::NoWarnings

FUNCTIONS

All functions below are intended to be called from the test script. They are all exported by default.

test ($name, $groupsub)

Executes $groupsub, which must be a reference to a subroutine, in a controlled environment and groups the results of all Test::Builder-style subtests launched inside into a single call to "ok" in Test::Builder, regardless of their number. If the test group is to be skipped (as discussed in "Skipping Groups"), calls "skip" in Test::Builder once instead.

In case the test group is not skipped, the first parameter to "ok" in Test::Builder and the value of the TODO string during same (see "TODO: BLOCK" in Test::More) are determined according to the following algorithm:

  1. if the test group terminates by throwing an exception, or terminates normally but without calling any subtest, it fails.

  2. otherwise, if any subtest failed outside of a TODO block, the group fails.

  3. otherwise, if any subtest succeeds inside of a TODO block, the group is flagged as an unexpected success.

  4. otherwise, if any subtest fails inside of a TODO block, the group results in a TODO (excused) failure.

  5. otherwise, the test group managed to avert all hazards and is a straight success (tada!!).

If any sub-tests failed in $groupsub, diagnostics will be propagated using "diag" in Test::Builder as usual.

The return value of test is 1 if the test group is a success (including a TODO unexpected success), 0 if it is a failure (including a TODO excused failure), and undef if the test group was skipped.

skip_next_tests ($number)

skip_next_tests ($number, $reason)

Skips the $number following groups of tests with reason $reason. Dies if we are currently skipping tests already.

skip_next_test ()

skip_next_test ($reason)

Equivalent to:

skip_next_tests 1;
skip_next_tests 1, $reason;

begin_skipping_tests ()

begin_skipping_tests;
begin_skipping_tests "reason";

Skips all subsequent groups of tests until blocked by "end_skipping_tests".

end_skipping_tests ()

Cancels the effect of "begin_skipping_tests". Has no effect if we are not currently skipping tests.

test_only ()

test_only "bla()", "reason";
test_only qr/^bla/;
test_only sub { /bla/ };

Skip all groups of tests whose name does not match the criteria. The criteria can be a plain string, a regular expression or a function.

test_only;

Resets to normal behavior.

PLUGIN FUNCTIONS

The following function relates to the plugin interface. It is not exported by default. See Test::Group::Extending for details.

next_test_plugin ($plugin)

Installs a plugin for the next test group. $plugin must be a subroutine reference.

CLASS METHODS

A handful of class methods are available to tweak the behavior of this module on a global basis. They are to be invoked like this:

Test::Group->foo(@args);

verbose ($level)

Sets verbosity level to $level, where 0 means quietest.

At level 1 and above there is a diagnostic line for the start of each test group.

At level 2 there is a diagnostic line showing the result of each subtest within top-level test groups. At level 3, the subtests of test groups nested within top level test groups also get diagnostic lines, and so on.

The default verbosity level is 0, or the value of the PERL_TEST_GROUP_VERBOSE environment variable if it is set.

catch_exceptions ()

Causes exceptions thrown from within the sub reference passed to "test" to be blocked; in this case, the test currently running will fail but the suite will proceed. This is the default behavior.

Note that catch_exceptions only deals with exceptions arising inside test blocks; those thrown by surrounding code (if any) still cause the test script to terminate as usual unless other appropriate steps are taken.

dont_catch_exceptions ()

Reverses the effect of "catch_exceptions", and causes exceptions thrown from a "test" sub reference to be fatal to the whole suite. This only takes effect for test subs that run after dont_catch_exceptions() returns; in other words this is not a whole-script pragma.

logfile ($classstate_logfile)

Sets the log file for caught exceptions to $classstate_logfile. From this point on, all exceptions thrown from within a text group (assuming they are caught, see "catch_exceptions") will be written to $classstate_logfile instead of being passed on to "diag" in Test::More. This is very convenient with exceptions with a huge text representation (say an instance of Error containing a stack trace).

use_subtest ()

This feature is experimental.

Causes Test::Group to use Test::Builder's subtest() feature as the test aggregation method, rather than doing black magic behind the scenes.

It is a fatal error to call use_subtest() if Test::Builder is too old to support subtests. To use subtests if they are available but fall back to normal operation if they are not, you can do:

eval { Test::Group->use_subtest };

Test::Group's exception handling mechanism is bypassed under use_subtest(), since Test::Builder::subtest() has its own exception handling system.

no_subtest ()

Turns off use_subtest.

BUGS

This class uses a somewhat unhealthy dose of black magic to take over control from Test::Builder when running inside a "test" group sub. While the temporary re-blessing trick used therein is thought to be very robust, it is not very elegant.

The experimental use_subtest() feature allows you to avoid the black magic if your Test::Builder is recent enough to support subtests.

SEE ALSO

Test::Simple, Test::More, Test::Builder, and friends

The perl-qa project, http://qa.perl.org/.

Similar modules on CPAN

Test::Class can be used to turn a test suite into a full-fledged object class of its own, in xUnit style. It also happens to support a similar form of test grouping using the :Test(no_plan) or :Tests attributes. Switching over to Test::Class will make a test suite more rugged and provide a number of advantages, but it will also dilute the "quick-and-dirty" aspect of .t files somewhat. This may or may not be what you want: for example, the author of this module enjoys programming most when writing tests, because the most infamous Perl hacks are par for the course then :-). Anyway TIMTOWTDI, and Test::Group is a way to reap some of the benefits of Test::Class (e.g. running only part of the test suite) without changing one's programming style too much.

AUTHORS

Nick Cleaton <ncleaton@cpan.org>

Dominique Quatravaux <domq@cpan.org>

Nicolas M. Thiéry <nthiery@users.sf.net>

LICENSE

Copyright (C) 2004 by IDEALX <http://www.idealx.com>

Copyright (c) 2009 by Nick Cleaton and Dominique Quatravaux

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.1 or, at your option, any later version of Perl 5 you may have available.