/ 
Games-Golf-0.14
/ Games::Golf::TestSuite

NAME

Games::Golf::TestSuite - Class that can run test suites

SYNOPSIS

use Games::Golf::TestSuite;

my $test = new Games::Golf::TestSuite( $hole );
$test->run ( @entries );

DESCRIPTION

A Perl Golfer, ensnared by the powerful desire to eliminate characters, avoids good programming practice. He uses, and abuses, any obscure functions and features that can be found. After much agony, he asks himself "Does it work?"

Without breaking his deep concentration, he wants to check whether his efforts have paid off - he needs a test script. Similarily, the game referee needs to know whether a random character sequence actually solves the problem. Writing a test script from scratch can be hard, but we have tackled the difficult tasks so you don't have to. A basic test suite could be written as:

# Does the solution compile?
$test->compile;

# Provide input and expect  output
$test->aioee("", "stdin", "stdout", "stderr", 0);

Could it get much easier than that? When the testing methods of this class are called, the test suite file is executed, and the results stored in the object attributes.

CONSTRUCTOR

new( $testcode, $entryfile )

The constructor reads the test suite code, and the name of the file that is supposed to store the entry code.

Any errors in compilation will result in an exception being raised.

Full details on how to create test suite files can be found in the UNNAMED section below.

LIMITS

Limits are restrictions on how a script can behave. For example, setting the stdout limit to 1024 will kill a script that tries writing more than 1kB of data to stdout.

By default, no restrictions are applied and the scripts being tested can do almost anything. This is rarely desirable when testing untrustworthy entries, so limits should be set whenever possible.

Unfortunately, the ability of limits to function correctly depends on the operating system used. Non-Unix systems may not support them particularly well - if at all. Users are strongly advised to read the section on PLATFORMS.

To retrieve or set limits the following accessor is used:

limit ( [OPTIONS] )

Pass no values to retrieve all limits and their values.

Pass one value to retrieve the values of a single limit.

Pass key/value pairs to set limits, previous values returned for convience.

The following limits can be set:

time => integer

Sets an overall time limit on how long a script can run for.

Platform dependent.

stdout => integer and/or stderr => integer

A data limit for stdout and stderr.

Platform dependent. (Win9X)

arch( [$arch] )

Select the underlying implementation for the capture() method. This is autoselected by Games::Golf::TestSuite, but bold users or testers can select the one they want. The method returns the name of the previously selected implementation.

With no argument given, it returns the currently selected implementation.

If one tries to select a non-existent implementation, it stays unchanged.

The capture() method can also decide to use a more appropriate implementation if it detects possible optimisations.

METHODS: RUNNING TEST-SUITES ON ENTRIES

These methods either test a single entry, or multiple entries via the test suite script. The test results are stored within the Games::Golf::Entry object, in the result attribute.

Exceptions may be raised in the event of an error, and will match /^check:/.

$test->check( ENTRY );

The test suite will be executed for a single Games::Golf::Entry object, unless the entry has been checked previously. To recheck, you must unset the result attribute of the Entry object.

The test result is returned (FIXME - what is the result?)

$test->run( [ ENTRIES ] );

Calls the check() method for each Games::Golf::Entry object passed. There is no return value.

METHODS: INDIVIDUAL TESTING

These tests are Games::Golf::TestSuite methods that hole makers can use in their hole test scripts.

Each of these methods will add to the results already stored in the Entry object being tested.

$test->compile;

This checks that the player script will compile.

The exit code from "perl -c" is returned. This could be used to short circuit full testing, if required. This function is subject to the security issues of the capture() function - including insecure arguments via filename.

$test->aioee( $args, $in, $out, $err, $exit );

Given $args and $in, the script should output $out on STDOUT, $err on STDERR and exit with code $exit. If you don't care about any of the three outputs (stdout, stderr or exit code), just pass "" or undef instead.

!!FIXME!! Rewrite required. !!FIXME!! Shouldn't aioee automatically set the limits? After all, when you give the expected output, the script should be killed whenever it prints more than the expected. On the other hand, a good error message should print all of the incorrect output.

Available tests for testing subs

If the hole requires a sub, the testsuite must first create the sub with makesub(), and then can use it.

The sub is stored in the sub attribute of the Games::Golf::TestSuite object, and is named after the hole, in the Games::Golf::TestSuite::Sandbox package (in case the sub calls itself).

$test->makesub;

Create a subroutine named after the hole in the sandbox. It can then be used through $test->sub( ... ) or by calling it directly by its name (the hole name).

This does count as a test in the testsuite. The sub should at least be valid Perl, shouldn't it?

$test->sub( ... );

Call the subroutine defined with makesub().

$test->ok( $result, $expected );

Similar to the ok() sub used in the Test module. The two parameters are in scalar context.

The following examples are straight from Test.pm:

$test->ok(0,1);             # failure: '0' ne '1'
$test->ok('broke','fixed'); # failure: 'broke' ne 'fixed'
$test->ok('fixed','fixed'); # success: 'fixed' eq 'fixed'
$test->ok('fixed',qr/x/);   # success: 'fixed' =~ qr/x/

$test->ok(sub { 1+1 }, 2);  # success: '2' eq '2'
$test->ok(sub { 1+1 }, 3);  # failure: '2' ne '3'
$test->ok(0, int(rand(2));  # (just kidding :-)

$test->ok 'segmentation fault', '/(?i)success/'; # regex match

Other test for the code itself

These tests are Games::Golf::TestSuite methods that hole makers can use in their hole test scripts.

The subtests use the code stored in attribute code, and update the attribute result.

$test->not_string( $s );

Test that the code in the entry doesn't contain the string $s.

$test->not_match( $regex );

Test that the code in the entry doesn't match $regex.

$test->not_op( $op );

Test that the code in the entry doesn't use the given opcode. (TODO)

ADVANCED METHODS

!!FIXME!! All of this should go at the end of the module file, after the documentation for TestSuite (aioee, compile, makesub, sub, ok). Should all this documentation appear in the module documentation?

Most users probably will never require these, but they are available if necessary.

$test->capture( $cmd, $args, $input )

The capture() method runs the $cmd program, with the $args command line arguments (given as a string !!FIXME!! does this always work?) and the content of $input is sent to its standard input.

capture() returns a list consisting of the script standard output, standard error and exit code.

This method is used internally by aioee() and compile().

Unix

This is our all bells and whistles capture function, and should be the safest for referees to use. At the moment it has many limitations and bugs.

Windows

Blurb about Windows security. (Oxymoron).

BUGS

Please report all bugs to:

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Games-Golf

TODO

Arrange for the entry script to be run with use strict; and warnings. (right now, they are run in a no strict, no warnings sandbox.)

AUTHORS

See Games::Golf or the AUTHORS file for the list of authors.

COPYRIGHT

This module is free software. It may be used, redistributed and/or modified under the same terms as Perl itself.

SEE ALSO

perl, Games::Golf, Games::Golf::Entry.