NAME
Fennec::Manual::User - The user manual for Fennec
DESCRIPTION
Covers usage of Fennec when writing tests and managing test suites.
CHAPTER 1 - INTRODUCTION
Fennec is a workflow based testing framework for Perl. It is designed to be easy to use, easy to extend, and to provide an incredible array of tools while staying out of your way. Fennec provides everything your used to in Perl testing framework along with enhancements inspired from other frameworks.
Fennec works just like traditional Test::Builder tools in that it allows you to write script style tests. Fennec takes this a step further by introducing behavior driven development with the concepts of test groups and workflows. Fennec also requires that each test file define a class which Fennec will instantiate.
CHAPTER 1.1 - WHY USE FENNEC?
Better tools lead to better tests, which in turn lead to better code. Perl has fallen behind the times when it comes to testing tools. Behavior driven development has proven it's worth in other laungages. A few projects have attempted to bring BDD to Perl, but they generally do not play well with existing Perl tools such as Test::Builder.
Fennec brings these new tools to Perl in a way that plays nicely with existing tools. Fennec will set you do things the way you already are, while allowing you to adopt new tools and features as you desire. Fennec also provides an excellent framework on which you can develop new tools that are compatible and re-usable.
CHAPTER 2 - USING FENNEC
CHAPTER 2.1 - WRITING A BASIC TEST CLASS
The most simple Fennec test class is a script that declares a package, uses Fennec::Standalone and makes assertions.
For example, t/my_test.t might look like:
#!/usr/bin/perl
package My::Test
use strict;
use warnings;
use Fennec::Standalone;
ok( 1, "1 is true, perl is not broken" );
done_testing();
*done_testing() is required at the end of all standalone tests.
CHAPTER 2.2 - ASSERTIONS
Fennec refers to test functions such as ok(), is(), and is_deeply() by the term 'assertion'. Using Fennec automatically gives you all the assertions normally provided by Test::More, Test::Warn, and Test::Exception. Please see Test::Simple if you are not already familiar with testing in Perl.
It is worth noting that these asserts are re-implemented the Fennec way. They do not use Test::Builder. This was done in order to provide better diagnostics and output. Test::Builder tools will work under Fennec, in most cases without any modification. It is also possible to tell Fennec to use the Test::Builder based tools instead of its own implementations, See Chapter 7.2.
Sample assertions:
ok( $should_be_true, "name of check" );
is( $have, $want, "name of other check" );
Fennec also allows for traditional assertions which die on failure, but only when tests are grouped. See Chapter 3.2 for more on groups.
tests my_tests {
die "check failed" unless $a == $b;
}
Using a Test::Builder library usually just works:
use Test::Thing qw( my_assert );
my_assert( ... );
CHAPTER 2.3 - DEFAULT ASSERTIONS
By default, Fennec will provide the following assert functions. See each referenced package for more details.
Fennec::Assert::Core::Simple
Fennec::Assert::Core::More
- is( $got, $want, $name )
- isnt($got, $want; $name)
- like($got, $want; $name)
- unlike($got, $want; $name)
- can_ok(*thing; @list )
- isa_ok(*thing; @list )
- is_deeply($got, $want; $name)
- advanced_is(got => $got, want => $want, name => $name, %options)
Fennec::Assert::Core::Exception
- lives_ok { ... } $name
- dies_ok { ... } $name
- throws_ok { ... } qr/.../, $name
- lives_and { ... } $name
Fennec::Assert::Core::Warn
- @list = capture_warnings { ... }
- warning_is { ... } $want, $name
- warnings_are { ... } \@want, $name
- warning_like { ... } qr/.../, $name
- warnings_like { ... } [ qr/.../, ... ], $name
- warnings_exist { ... } [ $string, $regex, ... ], $name
Fennec::Assert::Core::Package
- require_ok Package::Bareword
- require_ok( $package )
- use_ok Package::Bareword
- use_ok( $package, @args )
- use_into_ok( $package, $dest_package, @args )
Fennec::Assert::Core::Anonclass
CHAPTER 2.4 - CUSTOM INLINE ASSERTIONS
TODO - http://github.com/exodist/Fennec/issues/#issue/60
CHAPTER 2.5 - TODO AND SKIP
Fennec has the concept of todo tests, tests which are expected to fail. You can also mark groups as skip if they are really bad.
If an exception is thrown within a TODO
block or set, a failing TODO
result will be generated alerting you. This is a todo test and will not count as a failure in the grand scheme.
- Mark a test group as todo
-
# Failures will not count. tests not_yet_implemented => ( todo => "This will fail", method => sub { my $self = shift; ok( 0, "Hello world" ); }, );
- Skip a group completely
-
tests 'would die' => ( skip => "This will die", method => sub { my $self = shift; die( "I eat you" ); }, );
- Skip specific asserts
-
tests 'some pass' => sub { ok( 1, 'pass' ); TODO { ok( 0, 'fail' ); } "This will fail, I will fix it later"; }
CHAPTER 2.6 - META DATA
All Fennec test classes have an associated Fennec::TestFile::Meta object associated with them. A classes meta object can always be accessed with the $TEST_CLASS->fennec_meta() method. In addition the M() function can be used within the test class to access that classes meta object.
The meta object stores information you will almost never need to mess with. This includes data used by workflows, plugins, and other tools. NOTE The meta is associated with a class, not an object.
CHAPTER 2.7 - STASH
The stash is a hashref contained in the meta object. Because of this every test class has a single stash hash. You can always access the stash using the $TEST_CLASS->fennec_meta()->stash() method. In addition you can use the S() function from within a test class to quickly access and manipulate the stash.
- my $stash_ref = S();
- my %stash_copy = S();
-
The S() function returns a ref or hash pairs depending on the return context.
- my $value = S( $key )
-
When given a single key is provided as an argument, the value for that key will be returned.
- S( %MERGE_DATA )
-
When given one or more key/value pairs, the new data will be merged in with the old data.
- S( \%OVERRIDE )
-
When given a single hashref, that hashref will replace the entire stash.
CHAPTER 2.8 - PLAYING NICELY WITH PLUGINS AND EXTENTIONS
Many plugins and extentions rely on meta information being added to the stash. The most prevelant is the 'testing' key. This key should contain the name of the package being tested by the test class you are writing.
Example:
Package TEST::My::Module;
use strict;
use warnings;
use Fennec;
...
S( testing => 'My::Module' );
...
You should check with specific plugins for other stash keys that should be set.
CHAPTER 3 - SUITE MANAGEMENT
There are two ways to manage a test suite with Fennec. The simple way is to simply write .t files that use Fennec::Standalone. The other way is to turn control over to the Fennec 'runner' script.
CHAPTER 3.1 - STANDALONE FILES
Each Fennec test file is independent. Prove iterates the files for you running them in order. Every test has the overheard of re-loading Fennec and its tools. Files will not share a common configuration you must manually enter your configuration into each file.
Using Fennec::Standalone has already been introduced in Chapter 1.2.
CHAPTER 3.2 - SINGLE RUNNER
Fennec will be configured in t/Fennec.t, which will also serve as a runner. The runner will find all your test files and run them for you. The benefits of this system are reduced overhead, shared configuration, and the ability to potentially pre-load items that are cpu intensive to load.
The downside of a runner system is that it hijacks prove. Runners have been used in other Perl testing frameworks such as Test::Class. Some people have strong opinions about runners, usually unfavorable ones. Runners often have the problem that it is difficult to run a single test file.
Fennec has many command-line and editor integration tools that solve the most common complaints about runners. See Chapter 2.2.1 and Chapter 5 for more details.
To generate t/Fennec.t use the fennec_init
command. This will create the file with the default Fennec configuration. This file doubles as a configuration file. See Chapter 2.3 - Configuration for more details.
CHAPTER 3.2.1 - COMMAND LINE TOOLS
Fennec's command-line tools are intended to simplify working with Fennec.
- fennec_init
-
$ cd projectroot $ fennec_init $ ls t t/Fennec.t
This command-line tool will generate a default Fennec.t file under the t/ directory. This is where you start when writing a test-suite using fennec.
- fennec_prove
-
This only applies to managed tests. This command will not work for standalone tests.
$ fennec_prove - - [prove options] $ fennec_prove t/My/Test.pm - [prove options] $ fennec_prove t/My/Test.pm test_name [prove options] $ fennec_prove t/My/Test.pm line_number [prove options]
This is a wrapper around prove that lets you run a specific fennec test file, and optionally a specific test name or line number within that file. Running a specific line number is not 100% accurate, but pretty close.
- fennec_scaffold.pl
-
This will create test modules for all modules that do not currently have a test. It will find all modules under lib/ and create a test under t/. The created test file will have a single
require_ok( module_name )
test.Test names are assumed to follow the standard format. t/ will mirror lib/, and test module names will be the same as the module being tested. Package names will be the name of the tested package prefixed with
TEST::
. If you deviate from this style fennec_scaffold.pl will not find your test files and will assume they do not exist. - t/Fennec.t
-
This is the script that runs all your Fennec tests. It also doubles as a global configuration file. Most config options can be overridden using environment variables. See "Chapter 5 - Configuration" for more info.
- fennec_run.pl
-
This is identical to the default t/Fennec.t file. It can be used to run Fennec tests in a project without a t/Fennec.t.
CHAPTER 4 - TEST GROUPING
Test groups are methods containing assertions, typically defined using the tests
keyword. Tests will typically be grouped based on the feature or functionality being tested.
There are several benefits to grouping tests:
- Better structure leads to better readability
- Groups can be run in parrallel
- You can tell Fennec to run a specific group alone
CHAPTER 4.2 WRITING A TEST GROUP
The 'tests' keyword uses Devel::Declare to work similar to the 'sub' keyword, that is no semicolon is required after the definition. The keyword is further enhanced allowing for single or double quoted method names.
tests 'name of group' {
ok( 1, "name" );
...
}
tests bareword_name { ... }
tests "double quoted" { ... }
CHAPTER 5 - BEHAVIOR DRIVEN DEVELOPMENT
Behavior driven development or BDD is a way to structure tests in a more readable or manageable way than script style tests. The most well known example of BDD is Ruby's RSPEC, of which Fennec has an implementation.
Fennec implements BDD using the concept of workflows. Workflows are an abstraction of the common logic needed for BDD systems. Most Fennec BDD systems will be referred to generically as workflows.
When a Fennec test is run, it will first run all your package level assertions (ok, is, is_deeply, etc). While doing so it will also build and prepare your test groups and workflows. Once the building phase is complete, Fennec will instantiate your test class, and run all groups and workflows on that instance.
CHAPTER 5.1 - RSPEC
RSpec is a workflow originally created for Ruby. this manual will not go over RSpec in detail. Here is an example to get you started. It is worth noting that describe blocks are run as methods and get $self automatically.
describe 'test feature a' {
my $target;
before_each { $target = My::Class->new }
after_each { My::Class->reset_test_data }
# These will run in random order wrapped with the before and after:
it { ok( $target->feature_a( 'a' ), 'takes arg a' }
it { ok( $target->feature_a( 'b' ), 'takes arg b' }
it { ok( $target->feature_a( 'c' ), 'takes arg c' }
# You can nest describe methods
describe 'feature a other stuff' { ... }
}
The RSpec reference can be found here: http://rspec.info/
TODO - Need an RSpec guru to add more here.
CHAPTER 5.2 - CASES
Cases are a way to run the same assertions under multiple scenarios. In the following example each tests block will be run 4 times each with different values in var_a and var_b.
Note: Cases and tests can be define in any order. The resulting test groups will be run in random order, after all have been built.
cases 'test all combinations' {
my ( $var_a, $var_b );
# Scenarios
case set_a {( $var_a, $var_b ) = ( 5, 5 )}
case set_b {( $var_a, $var_b ) = ( 4, 6 )}
case set_c {( $var_a, $var_b ) = ( 3, 7 )}
case set_d {( $var_a, $var_b ) = ( 2, 8 )}
# Checks
tests 'equals 10' { is( $var_a + $var_b, 10, "sum is ten" )}
tests 'both vars are true' {
ok( $var_a, "var_a is true" );
ok( $var_b, "var_b is true" );
}
}
CHAPTER 5.3 - OBJECT METHODS
You can also define test groups, setups, and teardowns as methods on your test object.
sub setup_my_setup {
my $self = shift;
print "methods prefixed by setup_ will be run before tests defined as methods.";
}
sub test_method_as_test_by_prefix {
my $self = shift;
ok( 1, "methods prefixed by test_ will be run as method." );
}
sub teardown_my_teardown {
my $self = shift;
print "method prefixed by teardown_ will be run after tests defined as methods."
}
CHAPTER 6 - CONFIGURATION
Fennec is highly configurable. By default Fennec provides a sane configuration. The two main items to configure are the individual test files themselves, and the runner. When using Fennec::Standalone this distinction remains, however both are configured within each test file.
CHAPTER 6.1 - IN-TEST CONFIGURATION
These options are consistent between managed and standalone tests. These options are specified in test files when importing Fennec or Fennec::Standalone.
use Fennec OPTION => VALUE, ...;
# or
use Fennec::Standalone OPTION => VALUE, ...;
- random => $bool
-
Default is true. When true test groups and workflows will be run in random order. When false tests will run in hash key order, which is consistant though unpredictable.
- sort => $bool
-
Default is false. When true test groups and workflows will be run in sorted order.
- asserts => \@list
-
Used as a shortcut for loading additional assertion libraries. Each item should be a package name with the leading Fennec::Assert removed.
- workflows => \@list
-
Used as a shortcut for loading additional workflow libraries. Each item should be a package name with the leading Fennec::workflow removed.
- meta => { ... }
-
*YOU ALMOST CERTAINLY WILL NOT NEED TO SET THIS*
TODO - Fill this in
CHAPTER 6.2 - CONFIGURING THE RUNNER
- Managed Suite
-
For managed tests these configuration items should go in t/Fennec.t:
#!/usr/bin/perl ... Fennec::Runner->init( #### CONFIG OPTIONS GO HERE #### OPTION => VALUE, ... ); ...
- Standalone
-
For Standalone tests the runner is configured using an import option.
use Fennec::Standalone runner => { OPTION => VALUE, ... };
CHAPTER 6.2.1 - Runner Configuration Options
These are the runner configuration options, many of these do not make sense in a standalone test.
- parallel_files => $INTEGER
-
Default: 2
How many test files to run in parallel.
- parallel_tests => $INTEGER
-
Default: 2
How many test groups to run in parallel.
- default_asserts => \@LIST
-
Default: [ 'Core' ]
Assertion libraries to load by default. The leading Fennec::Assert may be omitted.
- default_workflows => \@LIST
-
Default: [qw/Spec Case Methods/],
Workflow libraries to load by default. The leading Fennec::Workflow may be omitted.
- load => [ [ $package1, @args1 ], [ $package2, @args2 ], ... ]
-
Should be an array of arrays. The inner arrays should contain a package name, followed by import arguments. Every package listed will be loaded into every test class.
Example:
load => [ [ qw/Package::A a b c /], [ qw/Package::B d e f /], ],
Is equvilent to adding the following to every single test file.
use Package::A qw/a b c/; use Package::B qw/d e f/;
- handlers => \@LIST
-
Default: [ 'TAP' ]
Output handlers to load by default. The leading Fennec::Handler may be omitted.
- collector => 'CLASS::NAME'
-
Default: 'Files'
Collector to use by default. The leading Fennec::Collector may be omitted.
- cull_delay => $SECONDS_FLOAT
-
Default: 0.01
How frequently the collector should check for new results. Lower is more frequently.
- filetypes => \@LIST
-
Default: [ 'Module' ]
File types to load by default. The leading Fennec::FileType may be omitted.
- ignore => \@LIST
-
Default: undef
List of filenames and regexes that specify files that should be ignored.
- root_workflow_class
-
Default: Fennec::Workflow
The root workflow class to which all workflows will be children.
- random => $BOOL
-
Default: TRUE
When true test groups and workflows will be run in random order. When false tests will run in hash key order, which is consistent though unpredictable.
CHAPTER 6.2.2 - Overriding Runner Options on The Command-Line
Almost every option above can be overridden on the command line using environment variables. Each items variable is its name in all caps prefixed by FENNEC_
.
Example:
$ FENNEC_PARRALLEL_FILES=5 FENNEC_PARRALLEL_TESTS=3 prove -I lib t/Fennec.t
CHAPTER 7 - PLUGINS
CHAPTER 7.1 - USEFUL PLUGINS
- Fennec::Template
-
Essentially Roles for Fennec. Lets you create template test groups and workflows that can be imported into any Fennec test class.
- Fennec::Server
-
UNDER DEVELOPMENT - Not stable or usable at the time of this writing.
A webserver will act as collector. You can monitor test status by visiting the server. There are also tools included for continuous integration testing of multiple projects.
CHAPTER 7.2 - GENERIC PLUGIN INSTRUCTIONS
Most plugins should be used in t/Fennec.t just after use Fennec::Runner. They may then require additional meta information be added to the stash.
#!/usr/bin/perl
use strict;
use warnings;
use Fennec::Runner;
use My::Plugin;
...
In standalone tests you want to use the plugin at the start of the file. Some may require you to use them before using Fennec::Standalone, others will require you do it afterwords, for most it probably does not matter.
CHAPTER 8 - EDITOR INTEGRATION
When editing a test file, it is useful to be able to run the test regularly on its own. With regular t/*.t files this is easy enough, however when test files are processed by a runner this can seem difficult. Fennec solves this problem through environment variables the editor can set. In act Fennec provides a mechanism to run specific test groups within a given file.
The Fennec runner script (t/Fennec.t) is designed to work well with editor integration. Almost every configuration option can be overridden via an environment variables (See Chapters 5.2.1-2). In addition to configuration variables, the script recognizes the following variables:
- FENNEC_SEED
-
When Fennec runs it prints the random seed that is used. This seed can be used with this variable to reproduce a previous run.
- FENNEC_FILE
-
Used to tell the runner to only run the specified file. Useful when working on a specific test and in editor integration.
- FENNEC_ITEM
-
Used to tell the runner to only run the specified item. Useful when working on a specific test and in editor integration.
The item can be either a group name, workflow name, or line number that defines either of those.
CHAPTER 8.1 - GENERIC EDITOR HOWTO
Editor integration generally means binding a key combination to run the current test file. With Fennec it means the same, with the addition of binding a second combination for running the current part of the current test file.
- Running the current file
-
Most editors that provide binding support can run a command and provide the name of the current file. You should bind your key to run the t/Fennec.t script while setting the FENNEC_FILE environment variable to the current test file.
Example Command:
FENNEC_FILE='%CURFILE' %PDIR/t/Fennec.t
- Running the current part of the current file
-
The traditional approach to this would be to train your editor to find the name of the section your editing and pass that to a command. The problem with that system is the existing and growing variety of ways to define groups and workflows. Fennec does support this system, but also provides a simpler way.
Internally Fennec attempts to track line numbers as groups and workflows are define. Since workflows are all defined before any are run, you can simply pass in a line number and Fennec will figure out the rest for you. Line number deduction is not 100% perfect, but is good enough.
The environment variable FENNEC_ITEM is used to provide either the line number or group name.
Example Command:
FENNEC_FILE='%CURFILE' FENNEC_ITEM='14' %PDIR/t/Fennec.t
CHAPTER 8.2 - VIM INTEGRATION
Fennec provides 2 files in the dist used for vim integration.
-
This file defines the
RunFennecLine()
function. This function runs the current test group in the current file. -
This file binds F10 to run the current file, and F8 to run the current part of the current file.
CHAPTER 8.3 - EMACS INTEGRATION
Currently there are no emacs users developing Fennec. We welcome anyone who would fill in this section.
CHAPTER 9 - TEST::BUILDER INFORMATION
Fennec does not use or depend on Test::Builder. However it has a module that allows Test::Builder based tools to work with Fennec. Fennec::Util::TBOverride overrides several parts of Test::Builder in order to send results to Fennec. It also provides a mechanism by which Test::Builder tools can be wrapped by Fennec to improve diagnostic output.
CHAPTER 9.1 - USING TEST::BUILDER TOOLS
Fennec::Util::TBOverride is loaded by default when Fennec loads. That means that in most cases you can simply import the Test::Builder tool and expect it to work fine. Fennec also provides wrappers around the most common Test::Builder tools for people that prefer them to the Fennec implementations.
CHAPTER 9.2 - USING TEST::BUILDER INSTEAD OF FENNEC ASSERTS
By default Fennec Loads the Core assertion library (Fennec::Assert::Core) which in turn loads the following (among others):
- Fennec::Assert::Core::Simple
- Fennec::Assert::Core::More
- Fennec::Assert::Core::Exception
- Fennec::Assert::Core::Warn
These are re-implementations of their Test::Builder namesakes. Fennec also provides a way to use the original implementations.
use Fennec asserts => ['TBCore'];
# or
use Fennec::Standalone asserts => ['TBCore'];
You can also do this globaly in t/Fennec.t:
#!/usr/bin/perl
...
'Fennec::Runner'->init(
default_asserts => ['TBCore'],
...
);
...
This will load Fennec::Assert::TBCore instead of Fennec::Assert::Core. Which in turn loads the following:
- Fennec::Assert::TBCore::Simple
- Fennec::Assert::TBCore::More
- Fennec::Assert::TBCore::Exception
- Fennec::Assert::TBCore::Warn
These are minimal wrappers around the actual Test::Builder based tools.
CHAPTER 10 - ADVANCED TOPICS
CHAPTER 10.1 - OUTPUT HANDLERS
By default Fennec produces TAP output, however it provides the tools to output any format you can think of. You can even output to multiple formats at once. Output plugins are called Handlers, they can be found in the Fennec::Handler:: namespace. See "Chapter 5 - Configuration" for the proper options to use.
CHAPTER 10.2 - COLLECTORS
Fennec allows for forking and parallelization, it needs a way to send results from the child processes to the parent. This is the role of the collector. Collector plugins can be found in the Fennec::Collector:: namespace. See "Chapter 5 - Configuration" for the proper options to use.
The default collector is based on temporary files. This collector is the most cross platform IPC available. A Socket based collector is in the works.
AUTHORS
Chad Granum exodist7@gmail.com
COPYRIGHT
Copyright (C) 2010 Chad Granum
Fennec is free software; Standard perl licence.
Fennec is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the license for more details.