/ 
Test2-Harness-2.000009
River stage two • 15 direct dependents • 20 total dependents
⭐ Starred 23 GitHub stars
/ README.md

NAME

Test2::Harness - AKA 'yath' Yet Another Test Harness, A modern alternative to prove.

DESCRIPTION

This is the primary documentation for the yath command, Test2::Harness, App::Yath and all other related components.

The yath command is an alternative to the prove command and Test::Harness.

PLATFORM SUPPORT

Test2::Harness/App::Yath is is focused on unix-like platforms. Most development happens on linux, but bsd, macos, etc should work fine as well.

Patches are welcome for any/all platforms, but the primary author (Chad 'Exodist' Granum) does not directly develop against non-unix platforms.

WINDOWS

Currently windows is not supported, and it is known that the package will not install on windows. Patches are be welcome, and it would be great if someone wanted to take on the windows-support role, but it is not a primary goal for the project.

QUICK START

You can use yath to run all the tests for your repo:

$ yath test

Some important notes for those used to the prove command:

COMMAND LINE HELP

There are a couple useful things to be aware of:

CONFIGURATION FILES

Yath will read from the following configuration files when you run it.

Note: These should be located in your projects root directory, yath will search parent directories to find them.

The format of the config files is:

# GLOBAL OPTIONS FOR ALL COMMANDS
-D/path/to/my/spcial/libs

[test]              # Options for the 'test' command
-j32                # Always use 32 processes for concurrency
-Irel(foo/bar)      # Always include this path relative to the location of the .rc file
-Irel(foo/bar/*)    # Wildcard expanded to multiple -I.. options

[start]
...

starts a comment

PRELOADING AND CONCURRENCY FOR FASTER TEST RUNS

You can preload modules that are expensive to load, then yath will launch tests from these preloaded states. In some cases this can provide a massive speedup:

yath test -PMoose -PList::Util -PScalar::Util

In addition yath can run multiple concurrent jobs (specified with the -j# command line option.

yath test -j16

You can combine these for a compounding performance boost:

yath test -j16 -PMoose

BENCHMARKING WITH THE MOOSE TEST SUITE:

As you can see concurrency and preloading make a huge difference in test run times!

See "ADVANCED CONCURRENCY" and "ADVANCED PRELOADING" for more information.

USING A WEB INTERFACE

Note: It is better to create a standalone yath web server, rather than creating a new instance for each run, Documentation on doing that will be linked here when it is written.

$ yath test --server

This will launch a web server (usually on http://127.0.0.1:8080, but the url will be printed for you at the start and end of the run) that allows you to view the results and filter/inspect the output in full detail.

NOTE: this requires installaction of one of the following sets of optional modules:

HARNESS DIRECTIVES INSIDE TESTS

yath will recognise a number of directive comments placed near the top of test files. These directives should be placed after the #! line but before any real code.

Real code is defined as any line that does not start with use, require, BEGIN, package, or #

HARNESS-NO-PRELOAD

#!/usr/bin/perl
# HARNESS-NO-PRELOAD

Use this if your test will fail when modules are preloaded. This will tell yath to start a new perl process to run the script instead of forking with preloaded modules.

Currently this implies HARNESS-NO-FORK, but that may not always be the case.

HARNESS-NO-FORK

#!/usr/bin/perl
# HARNESS-NO-FORK

Use this if your test file cannot run in a forked process, but instead must be run directly with a new perl process.

This implies HARNESS-NO-PRELOAD.

HARNESS-NO-STREAM

yath usually uses the Test2::Formatter::Stream formatter instead of TAP. Some tests depend on using a TAP formatter. This option will make yath use Test2::Formatter::TAP or Test::Builder::Formatter.

HARNESS-NO-IO-EVENTS

yath can be configured to use the Test2::Plugin::IOEvents plugin. This plugin replaces STDERR and STDOUT in your test with tied handles that fire off proper Test2::Event's when they are printed to. Most of the time this is not an issue, but any fancy tests or modules which do anything with STDERR or STDOUT other than print may have really messy errors.

Note: This plugin is disabled by default, so you only need this directive if you enable it globally but need to turn it back off for select tests.

HARNESS-NO-TIMEOUT

yath will usually kill a test if no events occur within a timeout (default 60 seconds). You can add this directive to tests that are expected to trip the timeout, but should be allowed to continue.

NOTE: you usually are doing the wrong thing if you need to set this. See: HARNESS-TIMEOUT-EVENT.

HARNESS-TIMEOUT-EVENT 60

yath can be told to alter the default event timeout from 60 seconds to another value. This is the recommended alternative to HARNESS-NO-TIMEOUT

HARNESS-TIMEOUT-POSTEXIT 15

yath can be told to alter the default POSTEXIT timeout from 15 seconds to another value.

Sometimes a test will fork producing output in the child while the parent is allowed to exit. In these cases we cannot rely on the original process exit to tell us when a test is complete. In cases where we have an exit, and partial output (assertions with no final plan, or a plan that has not been completed) we wait for a timeout period to see if any additional events come into

HARNESS-DURATION-LONG

This lets you tell yath that the test file is long-running. This is primarily used when concurrency is turned on in order to run longer tests earlier, and concurrently with shorter ones. There is also a yath option to skip all long tests.

This duration is set automatically if HARNESS-NO-TIMEOUT is set.

HARNESS-DURATION-MEDIUM

This lets you tell yath that the test is medium.

This is the default duration.

HARNESS-DURATION-SHORT

This lets you tell yath That the test is short.

HARNESS-CATEGORY-ISOLATION

This lets you tell yath that the test cannot be run concurrently with other tests. Yath will hold off and run these tests one at a time after all other tests.

HARNESS-CATEGORY-IMMISCIBLE

This lets you tell yath that the test cannot be run concurrently with other tests of this class. This is helpful when you have multiple tests which would otherwise have to be run sequentially at the end of the run.

Yath prioritizes running these tests above HARNESS-CATEGORY-LONG.

HARNESS-CATEGORY-GENERAL

This is the default category.

HARNESS-CONFLICTS-XXX

This lets you tell yath that no other test of type XXX can be run at the same time as this one. You are able to set multiple conflict types and yath will honor them.

XXX can be replaced with any type of your choosing.

NOTE: This directive does not alter the category of your test. You are free to mark the test with LONG or MEDIUM in addition to this marker.

HARNESS-JOB-SLOTS 2

HARNESS-JOB-SLOTS 1 10

Specify a range of job slots needed for the test to run. If set to a single value then the test will only run if it can have the specified number of slots. If given a range the test will require at least the lower number of slots, and use up to the maximum number of slots.

HARNESS-RETRY-n

This lets you specify a number (minimum n=1) of retries on test failure for a specific test. HARNESS-RETRY-1 means a failing test will be run twice and is equivalent to HARNESS-RETRY.

HARNESS-NO-RETRY

Use this to avoid this test being retried regardless of your retry settings.

ADVANCED CONCURRENCY

You can design your tests to use concurrency internally that is managed by yath!

You can sub-divide concurrency slots by specifying -j#:#. This will set the $ENV{T2_HARNESS_MY_JOB_CONCURRENCY} env var to the number of concurrency slots assigned to the test.

Note: Tests are normally only assigned 1 concurrency slot unless they have the # HARNESS-JOB-SLOTS MIN or #HARNESS-JOB-SLOTS MIN MAX headers at the top of the file (they can be after the shbang or use statements, but must be befor eany other code).

Here is an example of a test written to use anywhere from 1 to 5 slots depending on how much yath gives it. Implementation of wait() and start_child_process() is left to the reader.

#!/usr/bin/perl
use strict;
use warnings;
use Test2::V0;
use Test2::IPC;
# HARNESS-JOB-SLOTS 1 5

my @kids;

for my (1 .. $ENV{T2_HARNESS_MY_JOB_CONCURRENCY}) {
    push @kids => start_child_process(...);
}

wait($_) for @kids;

done_testing;

ADVANCED PRELOADING

You can create custom preload classes (still loaded with -PClass) that define advanced preload behavior:

package MyPreload;
use strict;
use warnings;

# Using this class will turn this into an advanced preload class
use Test2::Harness::Preload;

stage ONLY_MOOSE => sub {
    preload 'Moose';
};

stage ORGANIZATION_COMMON_MODULES => sub {
    eager();
    default();

    preload 'Moose';
    preload 'My::Common::Foo';
    preload 'My::Common::Bar';

    preload sub { ... };

    stage APP_A => sub {
        preload 'My::App::A';

        post_fork sub { do_this_before_each_test_after_fork() };
    };

    stage APP_B => sub {
        preload 'My::APP::B;
    };
};

1;

Custom preload classes use the Test2::Harness::Preload module to set up some key meta-information, then you can define preload "stages".

A "stage" is a process with a preloaded state waiting for tests to run, when it gets a test it will fork and the test will run in the fork. These stages/forks are smart and use goto::file to insure no stack frames or undesired state contamination will be introduced into your test process.

Stages may be nested, that is you can build a parent stage, then have child stages forked from that stage that inherit the state but also make independent changes.

There are also several hooks available to inject behavior pre-fork, post-fork and pre-launch.

FUNCTIONS AVAILABLE TO PRELOADS

SPECIFYING WHICH PRELOADS SHOULD BE USED FOR WHICH TESTS

The easiest way is to add a specific header comment to your test files:

#!/usr/bin/perl
use strict;
use warnings
# HARNESS-STAGE-MYSTAGE

You can also configure plugins (See "Plugins") to assign tests to stages.

PLUGINS

TODO: WRITE ME!

RENDERERS

TODO: WRITE ME!

RESOURCES

TODO: WRITE ME!

SOURCE

The source code repository for Test2-Harness can be found at http://github.com/Test-More/Test2-Harness/.

MAINTAINERS

AUTHORS

COPYRIGHT

Copyright Chad Granum exodist7@gmail.com.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See http://dev.perl.org/licenses/