NAME
Test::Snapshots - for testing stand alone scripts and executables
SYNOPSIS
use Test::More;
use Test::Snapshots;
test_all_snapshots('eg');Will go over all the .pl files in the eg/ directory, run them using with the content of the SCRIPT.out and SCRIPT.err files
Optional configurations before calling test_all_snapshots:
Test::Snapshots::debug(1);Get some extra diag messages
Test::Snapshots::combine(1);Combines the stdout and stderr and compares them to the SCRIPT.out file
Test::Snapshots::set_glob('*.t');Change the way we locate the scripts to be executed.
Test::Snapshots::set_accessories_dir('path/to/dir');Change the place where TS looks for .out files.
WARNING
This is alpha software. The API will most certainly change as the requirements get clearer.
Examples
Many of the unit test of this module are actually simple use cases with the files to be tested located in the eg/ subdirectory of the distribution. Check them out.
TODO
Add more test this module. Especially, we don't yet have failing tests.
Change the API to look more OO. Probably sg. like:
Test::Snapshots->set_glob() ->combine() ->set_accessories_dir() ->set_directories('eg') ->test_all_snapshots();
Allow subclassing or extending the module in some other way.
Deal with command line arguments. (.argv ?)
Deal with single file asseccories: A single file that holds the contents of the .in , .our, .err etc... file in sections.
E.g. the PHP core testing has .phpt files with sections:
--TEST-- Name of the test --FILE-- The code that needs to be saved in a file and executed --EXPECT-- The expected outputTest::Snapshots should be able to support that with the code to be executed inside as in the case of php or being outside as when testing executables.
Allow to pass several directories to traverse
Allow multiple runs in the same test script. (This will probably mean the test counting needs to be done separately or we will have to use the new "add plan" feature of Test::More.
Allow definiton of expected exit code for each file in some centralized form maybe similar to the way skip is defined.
Do we need a TODO test capability here?
Use Capture::Tiny ?
DESCRIPTION
Test::Snapshots was created especially to be able to test a large number of command line oriented executables. It does not matter if the executable is something compiled from C, a Perl, Python or PHP script.
Test::Snapshot can be seen as a very simple replacement of Expect. It will go over the designated direcory and run every execute like this:
executable arguments < input_file > output_file 2> error_fileIt will then check if the output_file is the same as the exepcted output file and if the error_file is the sameas the expected error file.
If an input file is not supplied then the < input_file part will be omitted.
The input file, the list of arguments and the expected output and error files all have the same name as the executable. So if you have an executable called fabricate.exe then you'd create the following files:
fabricate.exe.in
fabricate.exe.argv
fabricate.exe.out
fabricate.exe.err
fabricate.exe.exitIf .in is omitted we assume there is no input
If .argv is omitted then no arguments are provided
If .err or .out is omitted then it is assumed to be empty.
If .exit is omitted then it is expected that the exit code will be equal to the default exit code which is 0.
Multiple test cases
Sometime a single executable file should have multiple test cases. That is we might want to provide different .in and .argv files and expect different .out/.err/.exit values.
In order to allow such mode the files need to have a number in their name. So if you are testing xyz the files need to be
xyz.01.in
xyz.01.out
xyz.02.in
xyz.02.out
xyz.02.errThe expected number of test is the number of different numbers so if you have two files xyz.01.in and xyz.27.err then Test::Snapshots will run two test. One of them has no input and some expected error while the other has only input and not expected output or error.
Timeout
In order to avoid stuck test cases (e.g. waiting on STDIN) by default every test case can run up to 10 secs.
METHODS
timeout
Set timeout for the executions so if one of them gets stuck (e.g. waiting on STDIN) the whole test suit won't suffer.
Default 10 secs.
combine
Set to 1 if you'd like to combine the STDOUT and STDERR and compare the combined output to the .out file.
Default is 0 meaning they will be captured separatelly and compared separatelly to the .out and .err files.
set_glob
Set what glob to use to fine the files to be executed. Currently it defaults to '*.pl' but maybe it should have no default forcing the user to set one.
skip
Pass to it a hash ref of path => 'explanation' pairs for all the files that need to be skipped.
skip({
path => 'good reason',
path2 => 'some excuse',
});set_accessories_dir
We are calling the .out, .err etc files accessories.
In some cases you don't want them to be next to the script that are being tested. In such cases you can use the above function to tell Test::Snapshots where those files can be found.
command
By default Test::Snapshots will assume the files to be tested are stand alone executables or that at least they know where their interpreter is. So they will be executed directly.
In most of the cases you will want to run them with some specific command. e.g. You might want to make sure they run with the same perl interpreter as your test script runs. In that case call the following:
command($^X)In other cases the files need to be executed with some other tool, eg. the perl 6 or python interpreter which is in the path:
command("perl6");or
command("python");default_expected_exit_code
The exepceted exit code can be defined on a perl case basis in the .exit file. If the .exit file does not exist then there is a default expected exit code. Which is 0 by default.
Use this method to chane the default.
debug
You can turn on the debug flag by calling debug(1). If it is set Test::Snippets will call diag() with all kinds of information during the test execution.
test_all_snapshots
This is the call that actually goes out, locates all the files to be tested, sets the plan and executes all the test. Currently one should give a directory as a paramter to it but I plan to move that parameter to a separate method and to allow the setting of multiple directories.
test_single_file
Testing a single file. It gets the path to the file to be tested. The length of the prefix and optionally a case which is the 01, 02 etc. name of the test case for the multple-test-cases.
Currently this is considered an internal method.
See Also
Test::Simple, Test::More and Test::Most.
Test::Output, Capture::Tiny, Test::Cmd,
COPYRIGHT
Copyright 2009 Gabor Szabo gabor@szabgab.com http://szabgab.com/
LICENSE
This program is free software; you can redistribute it and/or modify it under the same terms as Perl 5 itself.
DISCLAIMER OF WARRANTY
BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENCE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.