/ 
Test-Aggregate-0.34_01
River stage one • 4 direct dependents • 5 total dependents
/ Test::Aggregate

NAME

Test::Aggregate - Aggregate *.t tests to make them run faster.

VERSION

Version 0.34_01

SYNOPSIS

use Test::Aggregate;

my $tests = Test::Aggregate->new( {
    dirs => $aggregate_test_dir,
} );
$tests->run;

DESCRIPTION

WARNING: this is ALPHA code. The interface is not guaranteed to be stable.

A common problem with many test suites is that they can take a long time to run. The longer they run, the less likely you are to run the tests. This module borrows a trick from Apache::Registry to load up your tests at once, create a separate package for each test and wraps each package in a method named run_the_tests. This allows us to load perl only once and related modules only once. If you have modules which are expensive to load, this can dramatically speed up a test suite.

USAGE

Create a separate directory for your tests. This should not be a subdirectory of your regular test directory. Write a small driver program and put it in your regular test directory (t/ is the standard):

use Test::Aggregate;
my $other_test_dir = 'aggregate_tests';
my $tests = Test::Aggregate->new( {
   dirs => $other_test_dir
});
$tests->run;

Take your simplest tests and move them, one by one, into the new test directory and keep running the Test::Aggregate program. You'll find some tests will not run in a shared environment like this. You can either fix the tests or simply leave them in your regular test directory. See how this distribution's tests are organized for an example.

Some tests cannot run in an aggregate environment. These may include test for this with the $ENV{TEST_AGGREGATE} variable:

package Some::Package;

BEGIN {
    die __PACKAGE__ ." cannot run in aggregated tests"
      if $ENV{TEST_AGGREGATE};
}

METHODS

new

my $tests = Test::Aggregate->new(
    {
        dirs            => 'aggtests',
        verbose         => 1,            # optional, but recommended
        dump            => 'dump.t',     # optional
        shuffle         => 1,            # optional
        matching        => qr/customer/, # optional
        set_filenames   => 0,            # optional and not recommended
        tidy            => 1,            # optional and experimental
        check_plan      => 1,            # optional and experimental
        test_nowarnings => 0,            # optional and experimental
    }
);

Creates a new Test::Aggregate instance. Accepts a hashref with the following keys:

  • dirs (mandatory)

    The directories to look in for the aggregated tests. This may be a scalar value of a single directory or an array refernce of multiple directories.

  • verbose (optional, but strongly recommended)

    If set with a true value, each test programs success or failure will be indicated with a diagnostic output. The output below means that aggtests/slow_load.t was an aggregated test which failed. This means it's much easier to determine which aggregated tests are causing problems.

    t/aggregate.........2/? 
#     ok - aggtests/boilerplate.t
#     ok - aggtests/00-load.t
# not ok - aggtests/subs.t
#     ok - aggtests/slow_load.t
t/aggregate.........ok
t/pod-coverage......ok
t/pod...............ok

    Note that three possible values are allowed for verbose:

    • 0 (default)

      No individual test program success or failure will be displayed.

    • 1

      Only failing test programs will have their failure status shown.

    • 2

      All test programs will have their success/failure shown.

  • dump (optional)

    You may list the name of a file to dump the aggregated tests to. This is useful if you have test failures and need to debug why the tests failed.

  • shuffle (optional)

    Ordinarily, the tests are sorted by name and run in that order. This allows you to run them in any order.

  • matching (optional)

    If supplied with a regular expression (requires the qr operator), will only run tests whose filename matches the regular expression.

  • set_filenames (optional)

    If supplied with a true value, this will cause the following to be added for each test:

    local $0 = $test_filename;

    This is the default behavior.

  • findbin (optional)

    If supplied with a true value, this will cause FindBin::again() to be called before each test file.

    This is turned off by default.

  • dry

    Just print the tests which will be run and the order they will be run in (obviously the order will be random if shuffle is true).

  • tidy

    If supplied a true value, attempts to run Perl::Tidy on the source code. This is a no-op if Perl::Tidy cannot be loaded. This option is experimental. Plus, if your tests are terribly convoluted, this could be slow and possibly buggy.

    If the value of this argument is the name of a file, assumes that this file is a .perltidyrc file.

  • check_plan

    If set to a true value, this will force Test::Aggregate to attempt to verify that any test which set a plan actually ran the correct number of tests. The code is rather tricky, so this is experimental.

  • test_nowarnings

    Disables Test::NoWarnings (fails if the module cannot be loaded). This is often used in conjunction with check_plan to subtract the extra test added by this module.

    This is experimental and somewhat problematic. Let me know if there are any problems.

run

$tests->run;

Attempts to aggregate and run all tests listed in the directories specified in the constructor.

SETUP/TEARDOWN

Since BEGIN and END blocks are for the entire aggregated tests and not for each test program (see CAVEATS), you might find that you need to have setup/teardown functions for tests. These are useful if you need to setup connections to test databases, clear out temp files, or any of a variety of tasks that your test suite might require. Here's a somewhat useless example, pulled from our tests:

#!/usr/bin/perl

use strict;
use warnings;

use lib 'lib', 't/lib';
use Test::Aggregate;
use Test::More;

my $dump = 'dump.t';

my ( $startup, $shutdown ) = ( 0, 0 );
my ( $setup,   $teardown ) = ( 0, 0 );
my $tests = Test::Aggregate->new(
    {
        dirs     => 'aggtests',
        dump     => $dump,
        startup  => sub { $startup++ },
        shutdown => sub { $shutdown++ },
        setup    => sub { $setup++ },
        teardown => sub { $teardown++ },
    }
);
$tests->run;
is $startup,  1, 'Startup should be called once';
is $shutdown, 1, '... as should shutdown';
is $setup,    4, 'Setup should be called once for each test program';
is $teardown, 4, '... as should teardown';

Note that you can still dump these to a dump file. This will only work if Data::Dump::Streamer 1.11 or later is installed.

There are four attributes which can be passed to the constructor, each of which expects a code reference:

  • startup

    startup => \&connect_to_database,

    This function will be called before any of the tests are run. It is not run in a BEGIN block.

  • shutdown

    shutdown => \&clean_up_temp_files,

    This function will be called after all of the tests are run. It will not be called in an END block.

  • setup

    setup => sub { 
   # this gets run before each test program.
},

    The setup function will be run before every test program.

  • teardown

    teardown => sub {
   # this gets run after every test program.
}

    The teardown function gets run after every test program.

CAVEATS

Not all tests can be included with this technique. If you have Test::Class tests, there is no need to run them with this. Otherwise:

  • __END__ and __DATA__ tokens.

    These won't work and the tests will call BAIL_OUT() if these tokens are seen.

  • BEGIN and END blocks.

    Since all of the tests are aggregated together, BEGIN and END blocks will be for the scope of the entire set of aggregated tests. If you need setup/teardown facilities, see "TEARDOWN" in SETUP.

  • Syntax errors

    Any syntax errors encountered will cause this program to BAIL_OUT(). This is why it's recommended that you move your tests into your new directory one at a time: it makes it easier to figure out which one has caused the problem.

  • no_plan

    Unfortunately, due to how this works, the plan is always no_plan. If Test::Builder implements "deferred plans", we can get a bit more safety. See http://groups.google.com/group/perl.qa/browse_thread/thread/d58c49db734844f4/cd18996391acc601?#cd18996391acc601 for more information. We now have an experimental 'check_plan' attribute to work around this.

  • Test::NoWarnings

    Great module. It loves to break aggregated tests since some might have warnings when others will not. You can disable it like this:

    my $tests = Test::Aggregate->new(
    dirs    => 'aggtests/',
    startup => sub { $INC{'Test/NoWarnings.pm'} = 1 },
);

    As an alternative, you can also disable it with:

    my $tests = Test::Aggregate->new({
   dirs            => 'aggtests',
   test_nowarnings => 0,
});

    This is needed when you use check_plan and have Test::NoWarnings used. This is because we do work internally to subtract the extra test added by Test::NoWarnings. It's painful and experimental. Good luck.

  • No 'skip_all' tests, please

    Tests which potentially 'skip_all' will cause the aggregate test suite to abort prematurely. Do not attempt to aggregate them. This may be fixed in a future release.

  • Variable "$x" will not stay shared at (eval ...

    Because each test is wrapped in a method call, any of your subs which access a variable in an outer scope will likely throw the above warning. Pass in arguments explicitly to suppress this.

    Instead of:

    my $x = 17;
sub foo {
    my $y = shift;
    return $y + $x;
}

    Write this:

    my $x = 17;
sub foo {
    my ( $y, $x ) = @_;
    return $y + $x;
}

  • Singletons

    Be very careful of code which loads singletons. Oftimes those singletons in test suites may be altered for testing purposes, but later attempts to use those singletons can fail dramatically as they're not expecting the alterations. (Your author has painfully learned this lesson with database connections).

DEBUGGING AGGREGATE TESTS

Before aggregating tests, make sure that you add tests one at a time to the aggregated test directory. Attempting to add many tests to the directory at once and then experiencing a failure means it will be much harder to track down which tests caused the failure.

Debugging aggregated tests which fail is a multi-step process. Let's say the following fails:

my $tests = Test::Aggregate->new(
    {
        dump    => 'dump.t',
        shuffle => 1,
        dirs    => 'aggtests',
    }
);
$tests->run;

Manually run the tests

The first step is to manually run all of the tests in the aggtests dir.

prove -r aggtests/

If the failures appear the same, fix them just like you would fix any other test failure and then rerun the Test::Aggregate code.

Sometimes this means that a different number of tests run from what the aggregted tests run. Look for code which ends the program prematurely, such as an exception or an exit statement.

Run a dump file

If this does not fix your problem, create a dump file by passing dump => $dumpfile to the constructor (as in the above example). Then try running this dumpfile directly to attempt to replicate the error:

prove -r $dumpfile

Tweaking the dump file

Assuming the error has been replicated, open up the dump file. The beginning of the dump file will have some code which overrides some Test::Builder internals. After that, you'll see the code which runs the tests. It will look similar to this:

if ( __FILE__ eq 'dump.t' ) {
    Test::More::diag("******** running tests for aggtests/boilerplate.t ********")
       if $ENV{TEST_VERBOSE};
    aggtestsboilerplatet->run_the_tests;

    Test::More::diag("******** running tests for aggtests/subs.t ********")
       if $ENV{TEST_VERBOSE};
    aggtestssubst->run_the_tests;

    Test::More::diag("******** running tests for aggtests/00-load.t ********")
       if $ENV{TEST_VERBOSE};
    aggtests00loadt->run_the_tests;

    Test::More::diag("******** running tests for aggtests/slow_load.t ********")
       if $ENV{TEST_VERBOSE};
    aggtestsslow_loadt->run_the_tests;
}

You can try to narrow down the problem by commenting out all of the run_the_tests lines and gradually reintroducing them until you can figure out which one is actually causing the failure.

COMMON PITFALLS

My Tests Through an Exception But Passed Anyway!

This really isn't a Test::Aggregate problem so much as a general Perl problem. For each test file, Test::Aggregate wraps the tests in an eval and checks my $error = $@. Unfortunately, we sometimes get code like this:

$server->ip_address('apple');

And internally, the 'Server' class throws an exception but uses its own evals in a DESTROY block (or something similar) to trap it. If the code you call uses an eval but fails to localize it, it wipes out your eval. Neat, eh? Thus, you never get a chance to see the error. For various reasons, this tends to impact Test::Aggregate when a DESTROY block is triggered and calls code which internally uses eval (e.g., DBIx::Class). You can often fix this with:

DESTROY {
   local $@ = $@;  # localize but preserve the value
   my $self = shift;
   # do whatever you want
}

BEGIN and END blocks

Remember that since the tests are now being run at once, these blocks will no longer run on a per-test basis, but will run for the entire aggregated set of tests. You may need to examine these individually to determine the problem.

CHECK and INIT blocks.

Sorry, but you can't use these (just as in modperl). See perlmod for more information about them and why they won't work.

Test::NoWarnings

This is a great test module. When aggregating tests together, however, it can cause pain as you'll often discover warnings that you never new existed. For a quick fix, add this before you attempt to run your tests:

$INC{'Test/NoWarnings.pm'} = 1;

That will disable Test::NoWarnings, but you'll want to go in later to fix them.

Paths

Many tests make assumptions about the paths to files and moving them into a new test directory can break this.

$0

Tests which use $0 can be problematic as the code is run in an eval through Test::Aggregate and $0 may not match expectations. This also means that it can behave differently if run directly from a dump file.

As it turns out, you can assign to $0! We do this by default and set the $0 to the correct filename. If you don't want this behavior, pass set_filenames => 0 to the constructor.

Minimal test case

If you cannot solve the problem, feel free to try and create a minimal test case and send it to me (assuming it's something I can run).

AUTHOR

Curtis Poe, <ovid at cpan.org>

BUGS

Please report any bugs or feature requests to bug-test-aggregate at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Aggregate. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc Test::Aggregate

You can also look for information at:

ACKNOWLEDGEMENTS

Many thanks to mauzo (http://use.perl.org/~mauzo/ for helping me find the 'skip_all' bug.

Thanks to Johan Lindström for pointing me to Apache::Registry.

COPYRIGHT & LICENSE

Copyright 2007 Curtis "Ovid" Poe, all rights reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

1 POD Error

The following errors were encountered while parsing the POD:

Around line 445:

Non-ASCII character seen before =encoding in 'Lindström'. Assuming UTF-8